[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:05:40 dist-test-slave-q06p krb5kdc[12002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:05:40 dist-test-slave-q06p krb5kdc[12002](info): set up 2 sockets
Sep 29 02:05:40 dist-test-slave-q06p krb5kdc[12002](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 02:05:41.331745 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:05:41 dist-test-slave-q06p krb5kdc[12002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:05:41.511998 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33045
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:33045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:41.848531 12015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:41.848846 12015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:41.848934 12015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:41.856274 12015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:05:41.856392 12015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:41.856438 12015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:41.856495 12015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:05:41.856539 12015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:05:41.866776 12015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:33045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:41.868144 12015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:41.869376 12015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:41.882184 12021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:41.882156 12020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:41.882406 12023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:41.883342 12015 server_base.cc:1034] running on GCE node
I20260929 02:05:41.884189 12015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:41.885690 12015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:41.886950 12015 hybrid_clock.cc:648] HybridClock initialized: now 1790647541886874 us; error 71 us; skew 500 ppm
I20260929 02:05:41.887313 12015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:41 dist-test-slave-q06p krb5kdc[12002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647541, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:41.890394 12015 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:05:41.892052 12015 webserver.cc:460] Webserver started at http://127.11.182.254:40361/ using document root <none> and password file <none>
I20260929 02:05:41.892758 12015 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:41.892907 12015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:41.893211 12015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:41.896291 12015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "081e8a76eff449b4b08f8bce97c5dcf6"
format_stamp: "Formatted at 2026-09-29 02:05:41 on dist-test-slave-q06p"
server_key: "650b256dcaf8c20ae10bee7e6e94df4b"
server_key_iv: "ba44ddfb98bb57988b408844ee3db414"
server_key_version: "encryptionkey@0"
I20260929 02:05:41.897117 12015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "081e8a76eff449b4b08f8bce97c5dcf6"
format_stamp: "Formatted at 2026-09-29 02:05:41 on dist-test-slave-q06p"
server_key: "650b256dcaf8c20ae10bee7e6e94df4b"
server_key_iv: "ba44ddfb98bb57988b408844ee3db414"
server_key_version: "encryptionkey@0"
I20260929 02:05:41.902910 12015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:05:41.906441 12030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:41.907944 12015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:05:41.908151 12015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "081e8a76eff449b4b08f8bce97c5dcf6"
format_stamp: "Formatted at 2026-09-29 02:05:41 on dist-test-slave-q06p"
server_key: "650b256dcaf8c20ae10bee7e6e94df4b"
server_key_iv: "ba44ddfb98bb57988b408844ee3db414"
server_key_version: "encryptionkey@0"
I20260929 02:05:41.908357 12015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:41.931442 12015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:41.936031 12015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:41.936339 12015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:41.956971 12015 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:33045
I20260929 02:05:41.957029 12081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:33045 every 8 connection(s)
I20260929 02:05:41.958386 12015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:05:41.960762 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 12015
I20260929 02:05:41.961102 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:05:41.963531 12082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:41.963660 11995 external_mini_cluster.cc:1428] Setting key 4f210f47e0d2e820cb21c45444bef561
I20260929 02:05:41.975626 12082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6: Bootstrap starting.
I20260929 02:05:41.979585 12082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:41.980984 12082 log.cc:824] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:05:41.984165 12082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6: No bootstrap required, opened a new log
I20260929 02:05:41.993289 12082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081e8a76eff449b4b08f8bce97c5dcf6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } }
I20260929 02:05:41.993693 12082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:41.993813 12082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081e8a76eff449b4b08f8bce97c5dcf6, State: Initialized, Role: FOLLOWER
Sep 29 02:05:41 dist-test-slave-q06p krb5kdc[12002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:41.994571 12082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081e8a76eff449b4b08f8bce97c5dcf6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } }
I20260929 02:05:41.994910 12082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:05:41.995066 12082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:05:41.995258 12082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:41.998096 12082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081e8a76eff449b4b08f8bce97c5dcf6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } }
I20260929 02:05:41.998642 12082 leader_election.cc:302] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 081e8a76eff449b4b08f8bce97c5dcf6; no voters:
I20260929 02:05:41.999321 12082 leader_election.cc:288] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:05:41.999423 12087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:05:42.000895 12087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [term 1 LEADER]: Becoming Leader. State: Replica: 081e8a76eff449b4b08f8bce97c5dcf6, State: Running, Role: LEADER
I20260929 02:05:42.001929 12087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081e8a76eff449b4b08f8bce97c5dcf6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } }
I20260929 02:05:42.002187 12082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:05:42.004607 12088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "081e8a76eff449b4b08f8bce97c5dcf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081e8a76eff449b4b08f8bce97c5dcf6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } } }
I20260929 02:05:42.004964 12088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:05:42.005422 12089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 081e8a76eff449b4b08f8bce97c5dcf6. Latest consensus state: current_term: 1 leader_uuid: "081e8a76eff449b4b08f8bce97c5dcf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081e8a76eff449b4b08f8bce97c5dcf6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } } }
I20260929 02:05:42.005698 12089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:05:42.006350 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:41.968394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.11.182.254:33045)
0929 02:05:41.969821 (+ 1427us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:41.969838 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:41.971181 (+ 1343us) server_negotiation.cc:407] Connection header received
0929 02:05:41.980441 (+ 9260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:41.980469 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:41.981011 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:41.981495 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:41.984686 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:41.986732 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:41.988445 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:41.989520 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:41.989727 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:41.995682 (+ 5955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:41.995690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:41.995705 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:41.995743 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:41.998828 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:42.000043 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:42.000054 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:42.000062 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:42.000271 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:42.000873 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:42.000882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:42.000884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:42.002093 (+ 1209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:42.002424 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:42.003929 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0929 02:05:42.004550 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1151,"thread_start_us":892,"threads_started":1}
I20260929 02:05:42.009209 12092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:05:42.015192 12092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:05:42.024458 12092 catalog_manager.cc:1347] Generated new cluster ID: c0b1dee4f9f84c66b8487326f1e402a3
I20260929 02:05:42.024591 12092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:05:42.037613 12092 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:05:42.039098 12092 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:05:42.054962 12092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 081e8a76eff449b4b08f8bce97c5dcf6: Generated new TSK 0
I20260929 02:05:42.056110 12092 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:05:42.103137 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33045
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:42.452507 12108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:42.452791 12108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:42.452884 12108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:42.460565 12108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:42.460686 12108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:42.460803 12108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:05:42.470825 12108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33045
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:42.472159 12108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:42.473430 12108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:42.487470 12113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:42.487471 12114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:42.488013 12116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:42.488750 12108 server_base.cc:1034] running on GCE node
I20260929 02:05:42.489382 12108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:42.490355 12108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:42.491583 12108 hybrid_clock.cc:648] HybridClock initialized: now 1790647542491543 us; error 64 us; skew 500 ppm
I20260929 02:05:42.491950 12108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:42 dist-test-slave-q06p krb5kdc[12002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:42.495491 12108 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:05:42.497022 12108 webserver.cc:460] Webserver started at http://127.11.182.193:42995/ using document root <none> and password file <none>
I20260929 02:05:42.497722 12108 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:42.497869 12108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:42.498183 12108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:42.501224 12108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "3c525a5a7be14742aaf0c0765c73bf41"
format_stamp: "Formatted at 2026-09-29 02:05:42 on dist-test-slave-q06p"
server_key: "87ae23d22913b2dc7ba95ac84540f27e"
server_key_iv: "9e7fb487154d50d2b2fb16c83268d600"
server_key_version: "encryptionkey@0"
I20260929 02:05:42.502014 12108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "3c525a5a7be14742aaf0c0765c73bf41"
format_stamp: "Formatted at 2026-09-29 02:05:42 on dist-test-slave-q06p"
server_key: "87ae23d22913b2dc7ba95ac84540f27e"
server_key_iv: "9e7fb487154d50d2b2fb16c83268d600"
server_key_version: "encryptionkey@0"
I20260929 02:05:42.507602 12108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 02:05:42.511543 12123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:42.513098 12108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:05:42.513290 12108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "3c525a5a7be14742aaf0c0765c73bf41"
format_stamp: "Formatted at 2026-09-29 02:05:42 on dist-test-slave-q06p"
server_key: "87ae23d22913b2dc7ba95ac84540f27e"
server_key_iv: "9e7fb487154d50d2b2fb16c83268d600"
server_key_version: "encryptionkey@0"
I20260929 02:05:42.513506 12108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:42.537891 12108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:42.542167 12108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:42.542469 12108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:42.543499 12108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:05:42.544866 12108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:05:42.544981 12108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:42.545135 12108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:05:42.545207 12108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:42.580869 12108 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:43097
I20260929 02:05:42.580904 12235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:43097 every 8 connection(s)
I20260929 02:05:42.582212 12108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:05:42.591331 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 12108
I20260929 02:05:42.591595 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:05:42.592195 11995 external_mini_cluster.cc:1428] Setting key ad8409f8033998f6518370e26f6ad854
Sep 29 02:05:42 dist-test-slave-q06p krb5kdc[12002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:42.605139 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:42.584766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:36261 (local address 127.11.182.254:33045)
0929 02:05:42.585195 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:42.585200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:42.586215 (+ 1015us) server_negotiation.cc:407] Connection header received
0929 02:05:42.588210 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:42.588215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:42.588307 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:42.588426 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:42.590095 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:42.591335 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:42.593177 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:42.593782 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:42.593918 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:42.598623 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:42.598628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:42.598632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:42.598670 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:42.601512 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:42.602486 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:42.602498 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:42.602500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:42.602649 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:42.603225 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:42.603230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:42.603235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:42.603573 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:42.603728 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:42.604527 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 02:05:42.604753 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":208,"threads_started":1}
I20260929 02:05:42.606218 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:42.585349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33045 (local address 127.11.182.193:36261)
0929 02:05:42.586051 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:42.586090 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:42.587905 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:42.588715 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:42.588724 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:42.589311 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:42.589915 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:42.589933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:42.591511 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:42.591515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:42.593020 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:42.593046 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:42.594029 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:42.594061 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:42.594388 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:42.595222 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:42.595257 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:42.598319 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:42.601730 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:42.601742 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:42.601769 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:42.602331 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:42.602822 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:42.602860 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:42.602872 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:42.603097 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:42.603725 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:42.603735 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:42.604116 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 02:05:42.604412 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 02:05:42.605006 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":278,"threads_started":1}
I20260929 02:05:42.607968 12236 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33045
I20260929 02:05:42.608327 12236 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:05:42.609182 12236 heartbeater.cc:511] Master 127.11.182.254:33045 requested a full tablet report, sending...
I20260929 02:05:42.611814 12047 ts_manager.cc:194] Registered new tserver with Master: 3c525a5a7be14742aaf0c0765c73bf41 (127.11.182.193:43097)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
I20260929 02:05:42.613755 12047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:36261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:05:42.638999 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33045
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:42.980027 12242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:42.980309 12242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:42.980473 12242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:42.987782 12242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:42.987911 12242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:42.988088 12242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:05:42.998395 12242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33045
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:42.999986 12242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:43.001417 12242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:43.015333 12247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:43.015256 12250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:43.015298 12248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:43.016819 12242 server_base.cc:1034] running on GCE node
I20260929 02:05:43.017638 12242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:43.018720 12242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:43.019997 12242 hybrid_clock.cc:648] HybridClock initialized: now 1790647543019929 us; error 57 us; skew 500 ppm
I20260929 02:05:43.020395 12242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:43 dist-test-slave-q06p krb5kdc[12002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:43.023758 12242 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:05:43.025465 12242 webserver.cc:460] Webserver started at http://127.11.182.194:33461/ using document root <none> and password file <none>
I20260929 02:05:43.026153 12242 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:43.026257 12242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:43.026531 12242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:43.029590 12242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "5ae0f1edac89437ea8fafe5c2e16b91b"
format_stamp: "Formatted at 2026-09-29 02:05:43 on dist-test-slave-q06p"
server_key: "110d9c5d951860a80f3b556ae411ceaf"
server_key_iv: "71a02f4e17240e27127941ee95c2261c"
server_key_version: "encryptionkey@0"
I20260929 02:05:43.030362 12242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "5ae0f1edac89437ea8fafe5c2e16b91b"
format_stamp: "Formatted at 2026-09-29 02:05:43 on dist-test-slave-q06p"
server_key: "110d9c5d951860a80f3b556ae411ceaf"
server_key_iv: "71a02f4e17240e27127941ee95c2261c"
server_key_version: "encryptionkey@0"
I20260929 02:05:43.035998 12242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:05:43.039713 12257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:43.041200 12242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:05:43.041355 12242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "5ae0f1edac89437ea8fafe5c2e16b91b"
format_stamp: "Formatted at 2026-09-29 02:05:43 on dist-test-slave-q06p"
server_key: "110d9c5d951860a80f3b556ae411ceaf"
server_key_iv: "71a02f4e17240e27127941ee95c2261c"
server_key_version: "encryptionkey@0"
I20260929 02:05:43.041529 12242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:43.063931 12242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:43.068783 12242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:43.069047 12242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:43.070003 12242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:05:43.071430 12242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:05:43.071506 12242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:43.071591 12242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:05:43.071637 12242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:43.108263 12242 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:35695
I20260929 02:05:43.108317 12369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:35695 every 8 connection(s)
I20260929 02:05:43.109601 12242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:05:43.116613 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 12242
I20260929 02:05:43.116839 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:05:43.117250 11995 external_mini_cluster.cc:1428] Setting key 3b27b677bf324a8225117f40ce3be485
Sep 29 02:05:43 dist-test-slave-q06p krb5kdc[12002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:43.132572 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:43.112626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:33871 (local address 127.11.182.254:33045)
0929 02:05:43.113111 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:43.113118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:43.114122 (+ 1004us) server_negotiation.cc:407] Connection header received
0929 02:05:43.116052 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:43.116058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:43.116147 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:43.116277 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:43.118036 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.119046 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.120738 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.121564 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.121715 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:43.125586 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:43.125591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:43.125594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:43.125627 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:43.128819 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.129916 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.129921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.129924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.130074 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.130659 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.130662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.130663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.130998 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:43.131105 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:43.132131 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0929 02:05:43.132307 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":220,"threads_started":1}
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:05:43.134346 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:43.113283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33045 (local address 127.11.182.194:33871)
0929 02:05:43.113965 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:43.114004 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:43.115770 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:43.116525 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:43.116533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:43.117143 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:43.117805 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:43.117834 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.119196 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.119203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:43.120580 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:43.120597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.121777 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.121782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:43.121998 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:43.122753 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:43.122776 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:43.125320 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:43.129001 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:43.129015 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:43.129039 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:43.129778 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:43.130201 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:43.130209 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:43.130216 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:43.130539 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:43.131146 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:43.131158 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:43.131716 (+ 558us) client_negotiation.cc:769] Sending connection context
0929 02:05:43.132073 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 02:05:43.132696 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":256,"threads_started":1}
I20260929 02:05:43.136121 12370 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33045
I20260929 02:05:43.136524 12370 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
I20260929 02:05:43.137454 12370 heartbeater.cc:511] Master 127.11.182.254:33045 requested a full tablet report, sending...
I20260929 02:05:43.139218 12047 ts_manager.cc:194] Registered new tserver with Master: 5ae0f1edac89437ea8fafe5c2e16b91b (127.11.182.194:35695)
I20260929 02:05:43.140450 12047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:33871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:05:43.162073 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33045
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:43.506991 12376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:43.507335 12376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:43.507501 12376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:43.516208 12376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:43.516386 12376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:43.516558 12376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:05:43.527439 12376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33045
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:43.528940 12376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:43.530248 12376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:43.545874 12381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:43.545711 12384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:43.545711 12382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:43.547003 12376 server_base.cc:1034] running on GCE node
I20260929 02:05:43.547616 12376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:43.548657 12376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:43.549893 12376 hybrid_clock.cc:648] HybridClock initialized: now 1790647543549847 us; error 37 us; skew 500 ppm
I20260929 02:05:43.550232 12376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:43 dist-test-slave-q06p krb5kdc[12002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:43.553341 12376 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:05:43.554967 12376 webserver.cc:460] Webserver started at http://127.11.182.195:33651/ using document root <none> and password file <none>
I20260929 02:05:43.555691 12376 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:43.555840 12376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:43.556160 12376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:43.559388 12376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "02bec3a7c18247eb899e2b36e00a67fd"
format_stamp: "Formatted at 2026-09-29 02:05:43 on dist-test-slave-q06p"
server_key: "d83b8dbd921c83135e955fc6b1b32053"
server_key_iv: "4daad386732672bd7240cf130e9ff39a"
server_key_version: "encryptionkey@0"
I20260929 02:05:43.560189 12376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "02bec3a7c18247eb899e2b36e00a67fd"
format_stamp: "Formatted at 2026-09-29 02:05:43 on dist-test-slave-q06p"
server_key: "d83b8dbd921c83135e955fc6b1b32053"
server_key_iv: "4daad386732672bd7240cf130e9ff39a"
server_key_version: "encryptionkey@0"
I20260929 02:05:43.565591 12376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 02:05:43.569824 12391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:43.571506 12376 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 02:05:43.571703 12376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "02bec3a7c18247eb899e2b36e00a67fd"
format_stamp: "Formatted at 2026-09-29 02:05:43 on dist-test-slave-q06p"
server_key: "d83b8dbd921c83135e955fc6b1b32053"
server_key_iv: "4daad386732672bd7240cf130e9ff39a"
server_key_version: "encryptionkey@0"
I20260929 02:05:43.571938 12376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:43.592409 12376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:43.596874 12376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:43.597204 12376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:43.598591 12376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:05:43.601123 12376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:05:43.601239 12376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:43.601370 12376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:05:43.601438 12376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:43.617939 12236 heartbeater.cc:503] Master 127.11.182.254:33045 was elected leader, sending a full tablet report...
I20260929 02:05:43.638679 12376 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:38919
I20260929 02:05:43.638724 12503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:38919 every 8 connection(s)
I20260929 02:05:43.640050 12376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:05:43.649780 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 12376
I20260929 02:05:43.650439 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:05:43.651027 11995 external_mini_cluster.cc:1428] Setting key f211a797b836a93974bf75ec9b990a79
Sep 29 02:05:43 dist-test-slave-q06p krb5kdc[12002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:43.661260 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:43.642687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:48671 (local address 127.11.182.254:33045)
0929 02:05:43.643165 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:43.643170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:43.644162 (+ 992us) server_negotiation.cc:407] Connection header received
0929 02:05:43.646120 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:43.646124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:43.646203 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:43.646300 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:43.647879 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.648613 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.650480 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.651087 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.651229 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:43.655151 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:43.655157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:43.655168 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:43.655200 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:43.658005 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.658815 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.658825 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.658827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.659000 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.659476 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.659481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.659484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.659771 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:43.659937 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:43.660654 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 02:05:43.660873 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":256,"threads_started":1}
I20260929 02:05:43.662309 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:43.643297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33045 (local address 127.11.182.195:48671)
0929 02:05:43.643994 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:43.644032 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:43.645774 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:43.646514 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:43.646522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:43.647132 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:43.647697 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:43.647714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.648746 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.648749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:43.650038 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:43.650050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.651293 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.651298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:43.651509 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:43.652186 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:43.652214 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:43.654919 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:43.658157 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:43.658167 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:43.658186 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:43.658684 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:43.659111 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:43.659116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:43.659120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:43.659356 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:43.659902 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:43.659912 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:43.660269 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 02:05:43.660547 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 02:05:43.661086 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":254,"threads_started":1}
I20260929 02:05:43.663869 12504 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33045
I20260929 02:05:43.664149 12504 heartbeater.cc:464] Registering TS with master...
I20260929 02:05:43.664880 12504 heartbeater.cc:511] Master 127.11.182.254:33045 requested a full tablet report, sending...
I20260929 02:05:43.666365 12047 ts_manager.cc:194] Registered new tserver with Master: 02bec3a7c18247eb899e2b36e00a67fd (127.11.182.195:38919)
I20260929 02:05:43.667412 12047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:48671
I20260929 02:05:43.668342 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:05:43 dist-test-slave-q06p krb5kdc[12002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 02:05:43.776055 11995 mini_kdc.cc:317] Time spent kinit for test-user: real 0.106s user 0.002s sys 0.007s
Sep 29 02:05:43 dist-test-slave-q06p krb5kdc[12002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:05:43.795740 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:43.778626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.11.182.193:43097)
0929 02:05:43.780062 (+ 1436us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:43.780074 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:43.780108 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:05:43.780183 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:43.780195 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:43.780482 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:43.780683 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:43.782141 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.783243 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.784931 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.785505 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.785603 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:43.789852 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:43.789867 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:43.789884 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:43.789928 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:43.793120 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.793807 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.793814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.793817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.793949 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.794309 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.794312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.794314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.794661 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:43.794969 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:43.795248 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 02:05:43.795458 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1236,"thread_start_us":1083,"threads_started":1}
W20260929 02:05:43.797178 12151 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41890
Sep 29 02:05:43 dist-test-slave-q06p krb5kdc[12002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:05:43 dist-test-slave-q06p krb5kdc[12002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:05:43.895082 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:43.879168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.11.182.193:43097)
0929 02:05:43.879413 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:43.879435 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:43.879735 (+ 300us) server_negotiation.cc:407] Connection header received
0929 02:05:43.879919 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:43.879925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:43.880057 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:43.880191 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:43.881851 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.882599 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.884696 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.885174 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.885285 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:43.889321 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:43.889327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:43.889330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:43.889367 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:43.892277 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.892981 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.892987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.892990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.893145 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.893623 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.893627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.893629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.893908 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:43.894098 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:43.894447 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:05:43.894690 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 02:05:43.896862 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 12108
I20260929 02:05:43.910964 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:43.897823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.11.182.193:43097)
0929 02:05:43.897985 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:43.897989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:43.898411 (+ 422us) server_negotiation.cc:407] Connection header received
0929 02:05:43.898528 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:43.898532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:43.898588 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:43.898668 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:43.899964 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.900656 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.901977 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:43.902419 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:43.902554 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:43.905441 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:43.905446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:43.905450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:43.905483 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:43.908059 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.908777 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.908789 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.908792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.908934 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:43.909313 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:43.909320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:43.909323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:43.909598 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:43.909854 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:43.910153 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 02:05:43.910368 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:05:43.911801 12231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:05:44.000996 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 12108
I20260929 02:05:44.018553 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 12242
Sep 29 02:05:44 dist-test-slave-q06p krb5kdc[12002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:05:44.035805 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:44.019645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.11.182.194:35695)
0929 02:05:44.020207 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:44.020214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:44.020244 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:05:44.020312 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:44.020318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:44.020545 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:44.020678 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:44.022069 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:44.023218 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:44.024572 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:44.025169 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:44.025343 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:44.029801 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:44.029818 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:44.029834 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:44.029873 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:44.032808 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:44.033513 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:44.033520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:44.033522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:44.033671 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:44.034152 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:44.034155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:44.034157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:44.034517 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:44.034747 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:44.035186 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 02:05:44.035391 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":1109,"threads_started":1}
I20260929 02:05:44.037173 12365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:05:44.126089 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 12242
I20260929 02:05:44.145525 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 12376
Sep 29 02:05:44 dist-test-slave-q06p krb5kdc[12002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:05:44.163861 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:44.146964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.11.182.195:38919)
0929 02:05:44.147715 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:44.147722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:44.147752 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:05:44.147814 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:44.147821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:44.148025 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:44.148167 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:44.149347 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:44.150380 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:44.152472 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:44.153074 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:44.153205 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:44.157645 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:44.157663 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:44.157680 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:44.157728 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:44.160913 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:44.161675 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:44.161683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:44.161686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:44.161842 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:44.162258 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:44.162263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:44.162265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:44.162661 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:44.163110 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:44.163359 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 02:05:44.163591 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":458,"threads_started":1}
I20260929 02:05:44.165114 12499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:05:44.251565 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 12376
I20260929 02:05:44.268939 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 12015
I20260929 02:05:44.270140 12077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:05:44.377813 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 12015
2026-09-29T02:05:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4167 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:05:46 dist-test-slave-q06p krb5kdc[12526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:05:46 dist-test-slave-q06p krb5kdc[12526](info): set up 2 sockets
Sep 29 02:05:46 dist-test-slave-q06p krb5kdc[12526](info): commencing operation
krb5kdc: starting...
W20260929 02:05:47.549278 11995 mini_kdc.cc:121] Time spent starting KDC: real 3.129s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 02:05:47.724395 11995 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.175s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 02:05:47.911454 11995 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.187s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 02:05:48.013301 11995 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.102s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/test-user.keytab.
W20260929 02:05:48.664976 11995 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.651s user 0.000s sys 0.004s
Sep 29 02:05:48 dist-test-slave-q06p krb5kdc[12526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
W20260929 02:05:49.115968 11995 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.11.182.254: real 0.406s user 0.000s sys 0.008s
I20260929 02:05:49.117008 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34143
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:34143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:49.436200 12539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:49.436474 12539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:49.436674 12539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:49.443753 12539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:05:49.443857 12539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:49.443964 12539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:49.444057 12539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:05:49.444137 12539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:05:49.453681 12539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:34143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:49.455024 12539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:49.456316 12539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:49.468143 12544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:49.468170 12547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:49.468114 12545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:49.469094 12539 server_base.cc:1034] running on GCE node
I20260929 02:05:49.469843 12539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:49.471223 12539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:49.472424 12539 hybrid_clock.cc:648] HybridClock initialized: now 1790647549472343 us; error 67 us; skew 500 ppm
I20260929 02:05:49.472759 12539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:49 dist-test-slave-q06p krb5kdc[12526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647549, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:49.475759 12539 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:05:49.477180 12539 webserver.cc:460] Webserver started at http://127.11.182.254:40241/ using document root <none> and password file <none>
I20260929 02:05:49.477795 12539 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:49.477890 12539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:49.478130 12539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:49.480978 12539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "a7040e1a1feb4d77b76552f3194381e5"
format_stamp: "Formatted at 2026-09-29 02:05:49 on dist-test-slave-q06p"
server_key: "1c0d44847a2813ee508abfee0594988b"
server_key_iv: "149e714aa53b4e1fc72ebe02b7fa5b6c"
server_key_version: "encryptionkey@0"
I20260929 02:05:49.481722 12539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "a7040e1a1feb4d77b76552f3194381e5"
format_stamp: "Formatted at 2026-09-29 02:05:49 on dist-test-slave-q06p"
server_key: "1c0d44847a2813ee508abfee0594988b"
server_key_iv: "149e714aa53b4e1fc72ebe02b7fa5b6c"
server_key_version: "encryptionkey@0"
I20260929 02:05:49.486727 12539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:05:49.490072 12554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:49.491474 12539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:05:49.491643 12539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "a7040e1a1feb4d77b76552f3194381e5"
format_stamp: "Formatted at 2026-09-29 02:05:49 on dist-test-slave-q06p"
server_key: "1c0d44847a2813ee508abfee0594988b"
server_key_iv: "149e714aa53b4e1fc72ebe02b7fa5b6c"
server_key_version: "encryptionkey@0"
I20260929 02:05:49.491822 12539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:49.522369 12539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:49.526611 12539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:49.526940 12539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:49.545634 12539 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:34143
I20260929 02:05:49.545653 12605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:34143 every 8 connection(s)
I20260929 02:05:49.547431 12539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:05:49.551167 12606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:49.553614 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 12539
I20260929 02:05:49.553833 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:05:49.554490 11995 external_mini_cluster.cc:1428] Setting key 36276eae500239c47aa095c42fbeb2a1
I20260929 02:05:49.563529 12606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5: Bootstrap starting.
I20260929 02:05:49.567385 12606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:49.568753 12606 log.cc:824] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:05:49 dist-test-slave-q06p krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:49.572113 12606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5: No bootstrap required, opened a new log
I20260929 02:05:49.578413 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:49.556585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.11.182.254:34143)
0929 02:05:49.557377 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:49.557389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:49.557431 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:05:49.559173 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:49.559196 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:49.559710 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:49.560056 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:49.561912 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:49.563268 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:49.565004 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:49.565690 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:49.565821 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:49.570726 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:49.570734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:49.570750 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:49.570785 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:49.573916 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:49.574731 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:49.574737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:49.574744 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:49.574986 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:49.575395 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:49.575399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:49.575401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:49.575976 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:49.576200 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:49.576843 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 02:05:49.577304 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":370,"threads_started":1}
I20260929 02:05:49.581027 12606 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7040e1a1feb4d77b76552f3194381e5" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34143 } }
I20260929 02:05:49.581423 12606 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:49.581539 12606 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7040e1a1feb4d77b76552f3194381e5, State: Initialized, Role: FOLLOWER
I20260929 02:05:49.582131 12606 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7040e1a1feb4d77b76552f3194381e5" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34143 } }
I20260929 02:05:49.582386 12606 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:05:49.582521 12606 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:05:49.582701 12606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:49.584785 12606 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7040e1a1feb4d77b76552f3194381e5" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34143 } }
I20260929 02:05:49.585336 12606 leader_election.cc:302] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7040e1a1feb4d77b76552f3194381e5; no voters:
I20260929 02:05:49.585932 12606 leader_election.cc:288] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:05:49.586063 12611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:05:49.586732 12611 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [term 1 LEADER]: Becoming Leader. State: Replica: a7040e1a1feb4d77b76552f3194381e5, State: Running, Role: LEADER
I20260929 02:05:49.587278 12611 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7040e1a1feb4d77b76552f3194381e5" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34143 } }
I20260929 02:05:49.587956 12606 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:05:49.590082 12613 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7040e1a1feb4d77b76552f3194381e5. Latest consensus state: current_term: 1 leader_uuid: "a7040e1a1feb4d77b76552f3194381e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7040e1a1feb4d77b76552f3194381e5" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34143 } } }
I20260929 02:05:49.590525 12613 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:05:49.590557 12612 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7040e1a1feb4d77b76552f3194381e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7040e1a1feb4d77b76552f3194381e5" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34143 } } }
I20260929 02:05:49.590902 12612 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:05:49.593036 12618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:05:49.598228 12618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:05:49.607492 12618 catalog_manager.cc:1347] Generated new cluster ID: 1cec4e3104b7449c846e37e199b06c5a
I20260929 02:05:49.607623 12618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:05:49.615629 12618 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:05:49.617002 12618 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:05:49.625624 12618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7040e1a1feb4d77b76552f3194381e5: Generated new TSK 0
I20260929 02:05:49.626747 12618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:05:49.678342 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34143
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:50.022357 12632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:50.022672 12632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:50.022771 12632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:50.030650 12632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:50.030784 12632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:50.030975 12632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:05:50.041294 12632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34143
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:50.042637 12632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:50.044093 12632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:50.058521 12638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:50.058583 12640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:50.058604 12637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:50.060405 12632 server_base.cc:1034] running on GCE node
I20260929 02:05:50.061216 12632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:50.062263 12632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:50.063512 12632 hybrid_clock.cc:648] HybridClock initialized: now 1790647550063468 us; error 38 us; skew 500 ppm
I20260929 02:05:50.063872 12632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:50 dist-test-slave-q06p krb5kdc[12526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:50.067713 12632 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:05:50.069399 12632 webserver.cc:460] Webserver started at http://127.11.182.193:34177/ using document root <none> and password file <none>
I20260929 02:05:50.070202 12632 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:50.070366 12632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:50.070680 12632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:50.073783 12632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "9ada4175dd6a4dcba12d764c7cab3efb"
format_stamp: "Formatted at 2026-09-29 02:05:50 on dist-test-slave-q06p"
server_key: "dc4deed69933f9bfbf3f1971d8bcb65a"
server_key_iv: "ba26b3aa05c0908bf3a16c274e2c524f"
server_key_version: "encryptionkey@0"
I20260929 02:05:50.074559 12632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "9ada4175dd6a4dcba12d764c7cab3efb"
format_stamp: "Formatted at 2026-09-29 02:05:50 on dist-test-slave-q06p"
server_key: "dc4deed69933f9bfbf3f1971d8bcb65a"
server_key_iv: "ba26b3aa05c0908bf3a16c274e2c524f"
server_key_version: "encryptionkey@0"
I20260929 02:05:50.080852 12632 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260929 02:05:50.084610 12647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:50.086175 12632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:05:50.086366 12632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "9ada4175dd6a4dcba12d764c7cab3efb"
format_stamp: "Formatted at 2026-09-29 02:05:50 on dist-test-slave-q06p"
server_key: "dc4deed69933f9bfbf3f1971d8bcb65a"
server_key_iv: "ba26b3aa05c0908bf3a16c274e2c524f"
server_key_version: "encryptionkey@0"
I20260929 02:05:50.086588 12632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:50.112210 12632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:50.117308 12632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:50.117705 12632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:50.118901 12632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:05:50.120486 12632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:05:50.120594 12632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:50.120734 12632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:05:50.120807 12632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:50.158864 12632 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:46067
I20260929 02:05:50.159030 12759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:46067 every 8 connection(s)
I20260929 02:05:50.160418 12632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:05:50.167542 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 12632
I20260929 02:05:50.167764 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:05:50.168383 11995 external_mini_cluster.cc:1428] Setting key f667c4fcb319d3959515335bf2969c70
Sep 29 02:05:50 dist-test-slave-q06p krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:50.182528 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:50.163594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:38055 (local address 127.11.182.254:34143)
0929 02:05:50.164049 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:50.164055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:50.165127 (+ 1072us) server_negotiation.cc:407] Connection header received
0929 02:05:50.167000 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:50.167004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:50.167106 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:50.167242 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:50.169178 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:50.170256 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:50.172040 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:50.172478 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:50.172622 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:50.176260 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:50.176265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:50.176268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:50.176299 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:50.179209 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:50.180086 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:50.180090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:50.180093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:50.180212 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:50.180678 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:50.180688 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:50.180690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:50.180958 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:50.181113 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:50.181908 (+ 795us) server_negotiation.cc:299] Negotiation successful
0929 02:05:50.182160 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260929 02:05:50.183632 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:50.164228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:34143 (local address 127.11.182.193:38055)
0929 02:05:50.164955 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:50.165005 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:50.166669 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:50.167499 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:50.167508 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:50.168171 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:50.168886 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:50.168933 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:50.170430 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:50.170441 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:50.171866 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:50.171879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:50.172741 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:50.172749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:50.173000 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:50.173743 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:50.173766 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:50.176017 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:50.179390 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:50.179401 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:50.179422 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:50.179917 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:50.180332 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:50.180337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:50.180341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:50.180553 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:50.181101 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:50.181111 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:50.181514 (+ 403us) client_negotiation.cc:769] Sending connection context
0929 02:05:50.181788 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 02:05:50.182388 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":281,"threads_started":1}
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:05:50.185318 12760 heartbeater.cc:347] Connected to a master server at 127.11.182.254:34143
I20260929 02:05:50.185595 12760 heartbeater.cc:464] Registering TS with master...
I20260929 02:05:50.186391 12760 heartbeater.cc:511] Master 127.11.182.254:34143 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
I20260929 02:05:50.188903 12571 ts_manager.cc:194] Registered new tserver with Master: 9ada4175dd6a4dcba12d764c7cab3efb (127.11.182.193:46067)
I20260929 02:05:50.190486 12571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:38055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:05:50.217422 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34143
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:50.563277 12766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:50.563617 12766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:50.563783 12766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:50.571385 12766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:50.571554 12766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:50.571727 12766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:05:50.582165 12766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34143
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:50.583639 12766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:50.585042 12766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:50.600075 12771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:50.600088 12772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:50.600732 12774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:50.601411 12766 server_base.cc:1034] running on GCE node
I20260929 02:05:50.602195 12766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:50.603413 12766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:50.604732 12766 hybrid_clock.cc:648] HybridClock initialized: now 1790647550604641 us; error 89 us; skew 500 ppm
I20260929 02:05:50.605438 12766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:50 dist-test-slave-q06p krb5kdc[12526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:50.613490 12766 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:05:50.615371 12766 webserver.cc:460] Webserver started at http://127.11.182.194:35825/ using document root <none> and password file <none>
I20260929 02:05:50.616250 12766 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:50.616408 12766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:50.616748 12766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:50.620038 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "5eea5935b753456eacc9e9d11019124a"
format_stamp: "Formatted at 2026-09-29 02:05:50 on dist-test-slave-q06p"
server_key: "569dfaf7562cd78852865fef676c3614"
server_key_iv: "fcf1aea63e7bc37a114905b3fb25b3d6"
server_key_version: "encryptionkey@0"
I20260929 02:05:50.621196 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "5eea5935b753456eacc9e9d11019124a"
format_stamp: "Formatted at 2026-09-29 02:05:50 on dist-test-slave-q06p"
server_key: "569dfaf7562cd78852865fef676c3614"
server_key_iv: "fcf1aea63e7bc37a114905b3fb25b3d6"
server_key_version: "encryptionkey@0"
I20260929 02:05:50.627305 12766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 02:05:50.631453 12781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:50.632987 12766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:05:50.633195 12766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "5eea5935b753456eacc9e9d11019124a"
format_stamp: "Formatted at 2026-09-29 02:05:50 on dist-test-slave-q06p"
server_key: "569dfaf7562cd78852865fef676c3614"
server_key_iv: "fcf1aea63e7bc37a114905b3fb25b3d6"
server_key_version: "encryptionkey@0"
I20260929 02:05:50.633446 12766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:50.656502 12766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:50.661655 12766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:50.662155 12766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:50.663390 12766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:05:50.664876 12766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:05:50.665004 12766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:50.665167 12766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:05:50.665241 12766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:50.706811 12766 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:44817
I20260929 02:05:50.706971 12893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:44817 every 8 connection(s)
I20260929 02:05:50.708259 12766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:05:50.718456 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 12766
I20260929 02:05:50.718688 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:05:50.719193 11995 external_mini_cluster.cc:1428] Setting key 7cb7d0dd7c06fda278ac75c54d461c3e
Sep 29 02:05:50 dist-test-slave-q06p krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:50.733103 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:50.711066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:35307 (local address 127.11.182.254:34143)
0929 02:05:50.711569 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:50.711576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:50.712948 (+ 1372us) server_negotiation.cc:407] Connection header received
0929 02:05:50.715069 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:50.715074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:50.715157 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:50.715279 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:50.717176 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:50.718267 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:50.720498 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:50.721056 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:50.721166 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:50.725734 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:50.725744 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:50.725753 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:50.725804 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:50.728623 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:50.729782 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:50.729786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:50.729789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:50.729954 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:50.730700 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:50.730704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:50.730706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:50.731165 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:50.731404 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:50.732504 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0929 02:05:50.732706 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":248,"threads_started":1}
I20260929 02:05:50.734689 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:50.711947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:34143 (local address 127.11.182.194:35307)
0929 02:05:50.712759 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:50.712803 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:50.714574 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:50.715588 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:50.715599 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:50.716214 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:50.716865 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:50.716884 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:50.718490 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:50.718495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:50.720256 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:50.720273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:50.721322 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:50.721332 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:50.721589 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:50.722442 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:50.722497 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:50.725400 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:50.728827 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:50.728842 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:50.728868 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:50.729590 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:50.730089 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:50.730097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:50.730103 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:50.730394 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:50.731383 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:50.731396 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:50.731958 (+ 562us) client_negotiation.cc:769] Sending connection context
0929 02:05:50.732311 (+ 353us) client_negotiation.cc:240] Negotiation successful
0929 02:05:50.732996 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":329,"threads_started":1}
I20260929 02:05:50.736505 12894 heartbeater.cc:347] Connected to a master server at 127.11.182.254:34143
I20260929 02:05:50.736905 12894 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:05:50.738291 12894 heartbeater.cc:511] Master 127.11.182.254:34143 requested a full tablet report, sending...
I20260929 02:05:50.740326 12571 ts_manager.cc:194] Registered new tserver with Master: 5eea5935b753456eacc9e9d11019124a (127.11.182.194:44817)
I20260929 02:05:50.741256 12571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:35307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:05:50.767292 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34143
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:51.142964 12900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:51.143311 12900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:51.143491 12900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:51.151093 12900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:51.151259 12900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:51.151451 12900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:05:51.161645 12900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34143
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:51.163069 12900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:51.164443 12900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:51.178709 12906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:51.178941 12905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:51.178721 12908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:51.179685 12900 server_base.cc:1034] running on GCE node
I20260929 02:05:51.180334 12900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:51.181473 12900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:51.182722 12900 hybrid_clock.cc:648] HybridClock initialized: now 1790647551182669 us; error 51 us; skew 500 ppm
I20260929 02:05:51.183105 12900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:51 dist-test-slave-q06p krb5kdc[12526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:51.186292 12900 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:05:51.187942 12900 webserver.cc:460] Webserver started at http://127.11.182.195:33021/ using document root <none> and password file <none>
I20260929 02:05:51.188802 12900 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:51.188913 12900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:51.189183 12900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:51.192654 12900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "cd7e2e8d0a3e4486ac395db0a259b07f"
format_stamp: "Formatted at 2026-09-29 02:05:51 on dist-test-slave-q06p"
server_key: "113400ce6c9065eb6fbc6f908a330a93"
server_key_iv: "db76a35aebe198ba98c368a329c430fa"
server_key_version: "encryptionkey@0"
I20260929 02:05:51.193504 12900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "cd7e2e8d0a3e4486ac395db0a259b07f"
format_stamp: "Formatted at 2026-09-29 02:05:51 on dist-test-slave-q06p"
server_key: "113400ce6c9065eb6fbc6f908a330a93"
server_key_iv: "db76a35aebe198ba98c368a329c430fa"
server_key_version: "encryptionkey@0"
I20260929 02:05:51.194065 12760 heartbeater.cc:503] Master 127.11.182.254:34143 was elected leader, sending a full tablet report...
I20260929 02:05:51.200019 12900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 02:05:51.203883 12915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:51.205550 12900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:05:51.205751 12900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "cd7e2e8d0a3e4486ac395db0a259b07f"
format_stamp: "Formatted at 2026-09-29 02:05:51 on dist-test-slave-q06p"
server_key: "113400ce6c9065eb6fbc6f908a330a93"
server_key_iv: "db76a35aebe198ba98c368a329c430fa"
server_key_version: "encryptionkey@0"
I20260929 02:05:51.205981 12900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:51.227713 12900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:51.232484 12900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:51.232818 12900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:51.233922 12900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:05:51.235385 12900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:05:51.235502 12900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:51.235647 12900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:05:51.235720 12900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:51.271237 12900 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:35941
I20260929 02:05:51.271291 13027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:35941 every 8 connection(s)
I20260929 02:05:51.273010 12900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:05:51.277484 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 12900
I20260929 02:05:51.277700 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:05:51.278124 11995 external_mini_cluster.cc:1428] Setting key 3b1e2ae446ba4fc1459645baa01920b9
Sep 29 02:05:51 dist-test-slave-q06p krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:51.296084 13030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.275808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:48533 (local address 127.11.182.254:34143)
0929 02:05:51.276328 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:51.276335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:51.277353 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 02:05:51.279652 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:51.279658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:51.279751 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:51.279919 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:51.282005 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.282977 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.284520 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.284932 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.285031 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.288940 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:51.288945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:51.288948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:51.288993 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:51.292513 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:51.293516 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:51.293522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:51.293526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:51.293690 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:51.294151 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:51.294156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:51.294160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:51.294430 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:51.294603 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:51.295428 (+ 825us) server_negotiation.cc:299] Negotiation successful
0929 02:05:51.295684 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":216,"threads_started":1}
I20260929 02:05:51.297237 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.276432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:34143 (local address 127.11.182.195:48533)
0929 02:05:51.277181 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:51.277220 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:51.279259 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:51.280225 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:51.280237 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:51.281065 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:51.281824 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.281847 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.283110 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.283114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.284362 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.284374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.285119 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.285124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.285325 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.286003 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:51.286026 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:51.288669 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:51.292722 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:51.292736 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:51.292752 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:51.293377 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:51.293812 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:51.293818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:51.293822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:51.294028 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:51.294576 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:51.294586 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:51.295042 (+ 456us) client_negotiation.cc:769] Sending connection context
0929 02:05:51.295336 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 02:05:51.295926 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":283,"threads_started":1}
I20260929 02:05:51.298789 13028 heartbeater.cc:347] Connected to a master server at 127.11.182.254:34143
I20260929 02:05:51.299214 13028 heartbeater.cc:464] Registering TS with master...
I20260929 02:05:51.300012 13028 heartbeater.cc:511] Master 127.11.182.254:34143 requested a full tablet report, sending...
I20260929 02:05:51.301564 12571 ts_manager.cc:194] Registered new tserver with Master: cd7e2e8d0a3e4486ac395db0a259b07f (127.11.182.195:35941)
I20260929 02:05:51.302323 12571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:48533
I20260929 02:05:51.306819 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:05:51.328533 13030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.313854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.11.182.254:34143)
0929 02:05:51.314101 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:51.314111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:51.314471 (+ 360us) server_negotiation.cc:407] Connection header received
0929 02:05:51.314607 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:51.314611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:51.314678 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:51.314764 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:51.316311 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.317442 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.319265 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.319971 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.320105 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.322966 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:51.322979 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:51.322983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:51.323018 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:51.325570 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:51.326367 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:51.326375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:51.326381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:51.326531 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:51.326967 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:51.326973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:51.326977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:51.327324 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:51.327590 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:51.327965 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 02:05:51.328190 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:05:51.335425 12571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:05:51.342593 12571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:05:51.367908 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.356271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:44817 (local address 127.0.0.1:53868)
0929 02:05:51.358467 (+ 2196us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:51.358489 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:51.358608 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:51.360134 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:51.360143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:51.360174 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:51.360530 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.360542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.362274 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.362281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.365055 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.365083 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.366675 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.366684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.366946 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.367002 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 02:05:51.367141 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 02:05:51.367409 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2053,"mutex_wait_us":52,"thread_start_us":2347,"threads_started":1}
I20260929 02:05:51.368994 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.357361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:35941 (local address 127.0.0.1:40134)
0929 02:05:51.359949 (+ 2588us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:51.359967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:51.360078 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:51.361022 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:51.361026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:51.361048 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:51.361393 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.361404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.363773 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.363778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.365907 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.365921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.368389 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.368393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.368516 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.368538 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:05:51.368602 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 02:05:51.368702 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2466,"mutex_wait_us":1,"thread_start_us":2018,"threads_started":1}
I20260929 02:05:51.369038 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.356812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.11.182.194:44817)
0929 02:05:51.359177 (+ 2365us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:51.359187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:51.359223 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:05:51.359337 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:51.359346 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:51.359639 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:51.359850 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:51.360748 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.362108 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.365215 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.366369 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.366562 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.366764 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:51.368451 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0929 02:05:51.368657 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2195,"thread_start_us":2039,"threads_started":1}
I20260929 02:05:51.369874 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.358187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.11.182.195:35941)
0929 02:05:51.359531 (+ 1344us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:51.359538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:51.360205 (+ 667us) server_negotiation.cc:407] Connection header received
0929 02:05:51.360305 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:51.360313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:51.360557 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:51.360733 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:51.361594 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.363610 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.366098 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.367666 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.367890 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.369403 (+ 1513us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:51.369491 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 02:05:51.369615 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":1335,"threads_started":1}
I20260929 02:05:51.370048 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.356190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:46067 (local address 127.0.0.1:58690)
0929 02:05:51.357543 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:51.357602 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:51.357809 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:51.362630 (+ 4821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:51.362634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:51.362658 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:51.363101 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.363119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.365070 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.365075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.367286 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.367299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.369370 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.369376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.369530 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.369559 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:05:51.369620 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 02:05:51.369735 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"thread_start_us":3541,"threads_started":1}
I20260929 02:05:51.370991 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.356462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.11.182.193:46067)
0929 02:05:51.361622 (+ 5160us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:51.361633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:51.361661 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:05:51.362173 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:51.362181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:51.362424 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:51.362638 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:51.363328 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.364899 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.367482 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.369141 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.370351 (+ 1210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.370495 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:51.370554 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 02:05:51.370661 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4972,"thread_start_us":4770,"threads_started":1}
I20260929 02:05:51.373867 12829 tablet_service.cc:1467] Processing CreateTablet for tablet f8205a099dc3422596c65dee8a005db7 (DEFAULT_TABLE table=test-table [id=9992ad17819a4214b218dfff48d23c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:05:51.374145 12963 tablet_service.cc:1467] Processing CreateTablet for tablet f8205a099dc3422596c65dee8a005db7 (DEFAULT_TABLE table=test-table [id=9992ad17819a4214b218dfff48d23c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:05:51.374405 12695 tablet_service.cc:1467] Processing CreateTablet for tablet f8205a099dc3422596c65dee8a005db7 (DEFAULT_TABLE table=test-table [id=9992ad17819a4214b218dfff48d23c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:05:51.375530 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8205a099dc3422596c65dee8a005db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:51.375530 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8205a099dc3422596c65dee8a005db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:51.375824 12695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8205a099dc3422596c65dee8a005db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:51.387908 13047 tablet_bootstrap.cc:492] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb: Bootstrap starting.
I20260929 02:05:51.391505 13048 tablet_bootstrap.cc:492] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a: Bootstrap starting.
I20260929 02:05:51.393030 13047 tablet_bootstrap.cc:654] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:51.393414 13049 tablet_bootstrap.cc:492] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f: Bootstrap starting.
I20260929 02:05:51.394624 13047 log.cc:824] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:05:51.396533 13048 tablet_bootstrap.cc:654] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:51.397087 13049 tablet_bootstrap.cc:654] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:51.398098 13049 log.cc:824] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:05:51.398293 13048 log.cc:824] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:05:51.399165 13047 tablet_bootstrap.cc:492] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb: No bootstrap required, opened a new log
I20260929 02:05:51.399595 13047 ts_tablet_manager.cc:1397] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 02:05:51.401551 13049 tablet_bootstrap.cc:492] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f: No bootstrap required, opened a new log
I20260929 02:05:51.401909 13049 ts_tablet_manager.cc:1397] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260929 02:05:51.402190 13048 tablet_bootstrap.cc:492] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a: No bootstrap required, opened a new log
I20260929 02:05:51.402473 13048 ts_tablet_manager.cc:1397] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260929 02:05:51.407691 13047 raft_consensus.cc:348] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } }
I20260929 02:05:51.408092 13047 raft_consensus.cc:374] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:51.408193 13047 raft_consensus.cc:729] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ada4175dd6a4dcba12d764c7cab3efb, State: Initialized, Role: FOLLOWER
I20260929 02:05:51.408877 13047 consensus_queue.cc:260] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } }
I20260929 02:05:51.411065 13047 ts_tablet_manager.cc:1428] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
W20260929 02:05:51.413130 12761 tablet.cc:2369] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:05:51.414196 13049 raft_consensus.cc:348] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } }
I20260929 02:05:51.414196 13048 raft_consensus.cc:348] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } }
I20260929 02:05:51.414661 13049 raft_consensus.cc:374] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:51.415156 13048 raft_consensus.cc:374] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:51.415218 13049 raft_consensus.cc:729] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7e2e8d0a3e4486ac395db0a259b07f, State: Initialized, Role: FOLLOWER
I20260929 02:05:51.415262 13048 raft_consensus.cc:729] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eea5935b753456eacc9e9d11019124a, State: Initialized, Role: FOLLOWER
I20260929 02:05:51.415946 13049 consensus_queue.cc:260] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } }
I20260929 02:05:51.415946 13048 consensus_queue.cc:260] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } }
I20260929 02:05:51.417248 13028 heartbeater.cc:503] Master 127.11.182.254:34143 was elected leader, sending a full tablet report...
I20260929 02:05:51.417771 12894 heartbeater.cc:503] Master 127.11.182.254:34143 was elected leader, sending a full tablet report...
I20260929 02:05:51.418154 13049 ts_tablet_manager.cc:1428] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 02:05:51.418355 13048 ts_tablet_manager.cc:1428] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a: Time spent starting tablet: real 0.016s user 0.008s sys 0.009s
W20260929 02:05:51.460598 12895 tablet.cc:2369] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:05:51.524765 13029 tablet.cc:2369] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:05:51.824770 13055 raft_consensus.cc:482] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:05:51.825160 13055 raft_consensus.cc:504] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } }
I20260929 02:05:51.827024 13055 leader_election.cc:288] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ada4175dd6a4dcba12d764c7cab3efb (127.11.182.193:46067), cd7e2e8d0a3e4486ac395db0a259b07f (127.11.182.195:35941)
I20260929 02:05:51.833803 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.827493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:38967 (local address 127.11.182.193:46067)
0929 02:05:51.827724 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:51.827730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:51.828407 (+ 677us) server_negotiation.cc:407] Connection header received
0929 02:05:51.828581 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:51.828587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:51.828660 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:51.828748 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:51.829368 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.830076 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.832047 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.832833 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.833025 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.833110 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:51.833398 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 02:05:51.833522 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:05:51.834223 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.827591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:41945 (local address 127.11.182.195:35941)
0929 02:05:51.827828 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:51.827835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:51.828602 (+ 767us) server_negotiation.cc:407] Connection header received
0929 02:05:51.828773 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:51.828781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:51.828867 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:51.828984 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:51.829576 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.830243 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.832077 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.832843 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.833024 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.833118 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:51.833428 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 02:05:51.833563 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 02:05:51.834367 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.827392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:46067 (local address 127.11.182.194:38967)
0929 02:05:51.828288 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:51.828304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:51.828442 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:51.828779 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:51.828783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:51.828815 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:51.829218 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.829230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.830215 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.830221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.831892 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.831906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.833074 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.833080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.833238 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.833262 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:05:51.833364 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 02:05:51.833499 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":42,"thread_start_us":540,"threads_started":1}
I20260929 02:05:51.834547 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.827417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:35941 (local address 127.11.182.194:41945)
0929 02:05:51.828449 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:51.828473 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:51.828632 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:51.829013 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:51.829018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:51.829042 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:51.829429 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.829438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.830373 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.830379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.831921 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:51.831940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.833127 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.833137 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:51.833277 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.833299 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:05:51.833403 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 02:05:51.833530 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":57,"thread_start_us":656,"threads_started":1}
I20260929 02:05:51.835562 12983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8205a099dc3422596c65dee8a005db7" candidate_uuid: "5eea5935b753456eacc9e9d11019124a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" is_pre_election: true
I20260929 02:05:51.835563 12715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8205a099dc3422596c65dee8a005db7" candidate_uuid: "5eea5935b753456eacc9e9d11019124a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" is_pre_election: true
I20260929 02:05:51.836053 12983 raft_consensus.cc:2454] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eea5935b753456eacc9e9d11019124a in term 0.
I20260929 02:05:51.836061 12715 raft_consensus.cc:2454] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eea5935b753456eacc9e9d11019124a in term 0.
I20260929 02:05:51.836762 12785 leader_election.cc:302] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5eea5935b753456eacc9e9d11019124a, 9ada4175dd6a4dcba12d764c7cab3efb, cd7e2e8d0a3e4486ac395db0a259b07f; no voters:
I20260929 02:05:51.837225 13055 raft_consensus.cc:2781] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:05:51.837370 13055 raft_consensus.cc:482] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:05:51.837499 13055 raft_consensus.cc:3037] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:51.839902 13055 raft_consensus.cc:504] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } }
I20260929 02:05:51.840515 13055 leader_election.cc:288] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [CANDIDATE]: Term 1 election: Requested vote from peers 9ada4175dd6a4dcba12d764c7cab3efb (127.11.182.193:46067), cd7e2e8d0a3e4486ac395db0a259b07f (127.11.182.195:35941)
I20260929 02:05:51.841214 12715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8205a099dc3422596c65dee8a005db7" candidate_uuid: "5eea5935b753456eacc9e9d11019124a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ada4175dd6a4dcba12d764c7cab3efb"
I20260929 02:05:51.841214 12983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8205a099dc3422596c65dee8a005db7" candidate_uuid: "5eea5935b753456eacc9e9d11019124a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f"
I20260929 02:05:51.841451 12983 raft_consensus.cc:3037] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:51.841439 12715 raft_consensus.cc:3037] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:51.844749 12983 raft_consensus.cc:2454] T f8205a099dc3422596c65dee8a005db7 P cd7e2e8d0a3e4486ac395db0a259b07f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea5935b753456eacc9e9d11019124a in term 1.
I20260929 02:05:51.844749 12715 raft_consensus.cc:2454] T f8205a099dc3422596c65dee8a005db7 P 9ada4175dd6a4dcba12d764c7cab3efb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea5935b753456eacc9e9d11019124a in term 1.
I20260929 02:05:51.845321 12785 leader_election.cc:302] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5eea5935b753456eacc9e9d11019124a, 9ada4175dd6a4dcba12d764c7cab3efb, cd7e2e8d0a3e4486ac395db0a259b07f; no voters:
I20260929 02:05:51.845724 13055 raft_consensus.cc:2781] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:05:51.846391 13055 raft_consensus.cc:686] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [term 1 LEADER]: Becoming Leader. State: Replica: 5eea5935b753456eacc9e9d11019124a, State: Running, Role: LEADER
I20260929 02:05:51.847254 13055 consensus_queue.cc:237] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } }
I20260929 02:05:51.852735 12570 catalog_manager.cc:5495] T f8205a099dc3422596c65dee8a005db7 P 5eea5935b753456eacc9e9d11019124a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5eea5935b753456eacc9e9d11019124a (127.11.182.194). New cstate: current_term: 1 leader_uuid: "5eea5935b753456eacc9e9d11019124a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada4175dd6a4dcba12d764c7cab3efb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eea5935b753456eacc9e9d11019124a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd7e2e8d0a3e4486ac395db0a259b07f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 35941 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:05:51 dist-test-slave-q06p krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:05:51.945554 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.930064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.11.182.193:46067)
0929 02:05:51.930284 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:51.930293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:51.930671 (+ 378us) server_negotiation.cc:407] Connection header received
0929 02:05:51.930939 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:51.930944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:51.931013 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:51.931122 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:51.932585 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.933333 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.934683 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.935192 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.935326 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.939864 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:51.939881 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:51.939897 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:51.939955 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:51.942688 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:51.943431 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:51.943440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:51.943443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:51.943589 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:51.943976 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:51.943981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:51.943984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:51.944348 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:51.944652 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:51.944942 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 02:05:51.945190 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 02:05:51 dist-test-slave-q06p krb5kdc[12526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:05:51 dist-test-slave-q06p krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:05:51.996347 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:51.979440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.11.182.193:46067)
0929 02:05:51.979683 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:51.979691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:51.980058 (+ 367us) server_negotiation.cc:407] Connection header received
0929 02:05:51.980277 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:51.980281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:51.980345 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:51.980447 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:51.981862 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.982608 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.983951 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:51.984389 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:51.984526 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:51.990231 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:51.990237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:51.990240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:51.990290 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:51.993498 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:51.994229 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:51.994236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:51.994240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:51.994385 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:51.994811 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:51.994819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:51.994822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:51.995156 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:51.995445 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:51.995716 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 02:05:51.995946 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 02:05:52 dist-test-slave-q06p krb5kdc[12526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:05:52 dist-test-slave-q06p krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:05:52.048223 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:52.031396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.11.182.193:46067)
0929 02:05:52.031752 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:52.031759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:52.031956 (+ 197us) server_negotiation.cc:407] Connection header received
0929 02:05:52.032171 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:52.032177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:52.032252 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:52.032369 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:52.033753 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:52.034496 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:52.035948 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:52.036539 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:52.036704 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:52.041646 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:52.041653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:52.041656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:52.041699 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:52.044770 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:52.045732 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:52.045737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:52.045740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:52.045983 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:52.046491 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:52.046496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:52.046499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:52.047013 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:52.047227 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:52.047687 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 02:05:52.047931 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 02:05:52.050189 12675 tablet_service.cc:2908] Rejecting scan request for tablet f8205a099dc3422596c65dee8a005db7: Uninitialized: clean time has not yet been initialized
I20260929 02:05:52.053726 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 12632
I20260929 02:05:52.069468 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:52.054731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58724 (local address 127.11.182.193:46067)
0929 02:05:52.055192 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:52.055198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:52.055557 (+ 359us) server_negotiation.cc:407] Connection header received
0929 02:05:52.055890 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:52.055899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:52.056033 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:52.056189 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:52.057498 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:52.058203 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:52.059782 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:52.060210 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:52.060367 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:52.062715 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:52.062721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:52.062724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:52.062761 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:52.066440 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:52.067399 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:52.067407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:52.067415 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:52.067553 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:52.067987 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:52.067991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:52.067993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:52.068275 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:52.068597 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:52.068897 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 02:05:52.069133 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260929 02:05:52.070241 12755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:05:52.161563 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 12632
I20260929 02:05:52.179666 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 12766
Sep 29 02:05:52 dist-test-slave-q06p krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:05:52.198184 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:52.180646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.11.182.194:44817)
0929 02:05:52.181103 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:52.181107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:52.181124 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:05:52.181180 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:52.181184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:52.181246 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:52.181334 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:52.182603 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:52.183491 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:52.185088 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:52.185498 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:52.185607 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:52.190669 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:52.190686 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:52.190703 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:52.190745 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:52.194386 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:52.195520 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:52.195530 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:52.195532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:52.195702 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:52.196281 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:52.196285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:52.196288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:52.196716 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:52.197004 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:52.197536 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 02:05:52.197749 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":223,"threads_started":1}
I20260929 02:05:52.199012 12889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:05:52.292670 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 12766
I20260929 02:05:52.311312 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 12900
Sep 29 02:05:52 dist-test-slave-q06p krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:05:52.329339 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:52.312139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.11.182.195:35941)
0929 02:05:52.312321 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:52.312325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:52.312363 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:05:52.312670 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:52.312674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:52.312751 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:52.312843 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:52.314176 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:52.315035 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:52.316524 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:52.317201 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:52.317334 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:52.322792 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:52.322817 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:52.322839 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:52.322921 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:52.326776 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:52.327645 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:52.327654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:52.327657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:52.327784 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:52.328093 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:52.328097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:52.328099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:52.328447 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:52.328665 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:52.328922 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 02:05:52.329087 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:05:52.330006 13023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:05:52.419935 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 12900
I20260929 02:05:52.437322 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 12539
I20260929 02:05:52.438588 12601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:05:52.548647 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 12539
2026-09-29T02:05:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (8167 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:05:52 dist-test-slave-q06p krb5kdc[13081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:05:52 dist-test-slave-q06p krb5kdc[13081](info): set up 2 sockets
Sep 29 02:05:52 dist-test-slave-q06p krb5kdc[13081](info): commencing operation
krb5kdc: starting...
W20260929 02:05:53.655745 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:05:53 dist-test-slave-q06p krb5kdc[13081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:05:53.847461 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:44461
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:44461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:54.183650 13094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:54.183980 13094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:54.184079 13094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:54.191620 13094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:05:54.191745 13094 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:05:54.191794 13094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:54.191847 13094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:54.191898 13094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:05:54.191960 13094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:05:54.202008 13094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:44461
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:44461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:54.203321 13094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:54.204547 13094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:54.218510 13100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:54.218667 13102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:54.218647 13099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:54.219681 13094 server_base.cc:1034] running on GCE node
I20260929 02:05:54.220582 13094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:54.222157 13094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:54.223418 13094 hybrid_clock.cc:648] HybridClock initialized: now 1790647554223352 us; error 45 us; skew 500 ppm
I20260929 02:05:54.223786 13094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:54 dist-test-slave-q06p krb5kdc[13081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:54.227834 13094 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:05:54.229689 13094 webserver.cc:460] Webserver started at http://127.11.182.254:33025/ using document root <none> and password file <none>
I20260929 02:05:54.230518 13094 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:54.230664 13094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:54.231081 13094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:54.234443 13094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90"
format_stamp: "Formatted at 2026-09-29 02:05:54 on dist-test-slave-q06p"
server_key: "fee465bbe3d780a10196ed7059355574"
server_key_iv: "47a17028d3071c382d646363e37e9356"
server_key_version: "encryptionkey@0"
I20260929 02:05:54.235361 13094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90"
format_stamp: "Formatted at 2026-09-29 02:05:54 on dist-test-slave-q06p"
server_key: "fee465bbe3d780a10196ed7059355574"
server_key_iv: "47a17028d3071c382d646363e37e9356"
server_key_version: "encryptionkey@0"
I20260929 02:05:54.241973 13094 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 02:05:54.246232 13109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:54.248436 13094 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:05:54.248694 13094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90"
format_stamp: "Formatted at 2026-09-29 02:05:54 on dist-test-slave-q06p"
server_key: "fee465bbe3d780a10196ed7059355574"
server_key_iv: "47a17028d3071c382d646363e37e9356"
server_key_version: "encryptionkey@0"
I20260929 02:05:54.248963 13094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:54.278172 13094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:54.283890 13094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:54.284497 13094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:54.312287 13094 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:44461
I20260929 02:05:54.312304 13170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:44461 every 8 connection(s)
I20260929 02:05:54.313822 13094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:05:54.318635 13171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:54.320577 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 13094
I20260929 02:05:54.320838 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:05:54.321646 11995 external_mini_cluster.cc:1428] Setting key d4ce4f91c9fdaa8b2bbcc75a731f7f5e
I20260929 02:05:54.336413 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90: Bootstrap starting.
Sep 29 02:05:54 dist-test-slave-q06p krb5kdc[13081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:54.340372 13171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:54.341992 13171 log.cc:824] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90: Log is configured to *not* fsync() on all Append() calls
I20260929 02:05:54.346001 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90: No bootstrap required, opened a new log
I20260929 02:05:54.350440 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:54.324442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.11.182.254:44461)
0929 02:05:54.325529 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:54.325544 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:54.325596 (+ 52us) server_negotiation.cc:407] Connection header received
0929 02:05:54.327628 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:54.327657 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:54.328189 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:54.328659 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:54.330232 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.332333 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.334481 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.335720 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.335996 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:54.340653 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:54.340665 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:54.340688 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:54.340757 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:54.344932 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:54.345879 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:54.345888 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:54.345897 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:54.346155 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:54.346768 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:54.346772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:54.346775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:54.347654 (+ 879us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:54.347918 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:54.348579 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 02:05:54.349137 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":565,"threads_started":1}
I20260929 02:05:54.355402 13171 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44461 } }
I20260929 02:05:54.355796 13171 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:54.355930 13171 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2ea2ec89d14a5fb0259ff2c4d52c90, State: Initialized, Role: FOLLOWER
I20260929 02:05:54.356581 13171 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44461 } }
I20260929 02:05:54.356830 13171 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:05:54.356973 13171 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:05:54.357159 13171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:54.360000 13171 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44461 } }
I20260929 02:05:54.360857 13171 leader_election.cc:302] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe2ea2ec89d14a5fb0259ff2c4d52c90; no voters:
I20260929 02:05:54.361824 13171 leader_election.cc:288] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:05:54.361956 13176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:05:54.362826 13176 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [term 1 LEADER]: Becoming Leader. State: Replica: fe2ea2ec89d14a5fb0259ff2c4d52c90, State: Running, Role: LEADER
I20260929 02:05:54.363468 13176 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44461 } }
I20260929 02:05:54.364482 13171 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:05:54.366884 13177 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44461 } } }
I20260929 02:05:54.367486 13177 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [sys.catalog]: This master's current role is: LEADER
I20260929 02:05:54.368165 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2ea2ec89d14a5fb0259ff2c4d52c90. Latest consensus state: current_term: 1 leader_uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2ea2ec89d14a5fb0259ff2c4d52c90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44461 } } }
I20260929 02:05:54.368503 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90 [sys.catalog]: This master's current role is: LEADER
I20260929 02:05:54.374497 13184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:05:54.379990 13184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:05:54.389675 13184 catalog_manager.cc:1347] Generated new cluster ID: b9d97bc2e24145f2bb7a5a1cba9932a1
I20260929 02:05:54.389840 13184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:05:54.403646 13184 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:05:54.405104 13184 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:05:54.417021 13184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90: Generated new TSK 0
I20260929 02:05:54.418219 13184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:05:54.475914 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:44461
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:54.841061 13197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:54.841638 13197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:54.841796 13197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:54.849488 13197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:54.849622 13197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:54.849790 13197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:05:54.849915 13197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:05:54.862443 13197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.182.254:44461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:54.863924 13197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:54.865541 13197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:54.880729 13205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:54.880757 13203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:54.881347 13202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:54.882450 13197 server_base.cc:1034] running on GCE node
I20260929 02:05:54.883322 13197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:54.884476 13197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:54.885800 13197 hybrid_clock.cc:648] HybridClock initialized: now 1790647554885722 us; error 62 us; skew 500 ppm
I20260929 02:05:54.886212 13197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:54 dist-test-slave-q06p krb5kdc[13081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:54.889775 13197 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:05:54.892102 13197 webserver.cc:460] Webserver started at http://127.11.182.193:42705/ using document root <none> and password file <none>
I20260929 02:05:54.892951 13197 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:54.893095 13197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:54.893466 13197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:54.896699 13197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "348a62b8c67e465f93ba8a5500720d10"
format_stamp: "Formatted at 2026-09-29 02:05:54 on dist-test-slave-q06p"
server_key: "1d55094dda38eb2ed9e6c31d6406d4ae"
server_key_iv: "50e29c5434d9099b24c10f82ec5ae815"
server_key_version: "encryptionkey@0"
I20260929 02:05:54.897580 13197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "348a62b8c67e465f93ba8a5500720d10"
format_stamp: "Formatted at 2026-09-29 02:05:54 on dist-test-slave-q06p"
server_key: "1d55094dda38eb2ed9e6c31d6406d4ae"
server_key_iv: "50e29c5434d9099b24c10f82ec5ae815"
server_key_version: "encryptionkey@0"
I20260929 02:05:54.903714 13197 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 02:05:54.907662 13212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:54.909580 13197 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:05:54.910085 13197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "348a62b8c67e465f93ba8a5500720d10"
format_stamp: "Formatted at 2026-09-29 02:05:54 on dist-test-slave-q06p"
server_key: "1d55094dda38eb2ed9e6c31d6406d4ae"
server_key_iv: "50e29c5434d9099b24c10f82ec5ae815"
server_key_version: "encryptionkey@0"
I20260929 02:05:54.910359 13197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:54.924458 13197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:54.929072 13197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:54.929380 13197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:54.932408 13197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:05:54.932531 13197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:54.932693 13197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:05:54.932761 13197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:05:54 dist-test-slave-q06p krb5kdc[13081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:54.957753 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:54.932945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:46641 (local address 127.11.182.254:44461)
0929 02:05:54.933812 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:54.933817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:54.934637 (+ 820us) server_negotiation.cc:407] Connection header received
0929 02:05:54.936842 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:54.936849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:54.937009 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:54.937191 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:54.939684 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.940738 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.943674 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.944083 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.944232 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:54.949332 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:54.949343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:54.949351 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:54.949413 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:54.953765 (+ 4352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:54.954625 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:54.954634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:54.954640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:54.954838 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:54.955339 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:54.955347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:54.955352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:54.955836 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:54.956029 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:54.956889 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 02:05:54.957209 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":665,"threads_started":1}
I20260929 02:05:54.958997 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:54.933656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44461 (local address 127.11.182.193:46641)
0929 02:05:54.934475 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:54.934515 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:54.936458 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:54.937465 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:54.937474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:54.938297 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:54.939450 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:54.939485 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.940975 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.940980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:54.943496 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:54.943510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.944456 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.944474 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:54.944705 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:54.945689 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:54.945715 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:54.948995 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:54.953951 (+ 4956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:54.953967 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:54.953982 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:54.954476 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:54.954990 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:54.955000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:54.955005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:54.955201 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:54.956019 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:54.956026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:54.956432 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 02:05:54.956688 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 02:05:54.957127 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":349,"threads_started":1}
I20260929 02:05:54.988416 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:54.970292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:41037 (local address 127.11.182.254:44461)
0929 02:05:54.970541 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:54.970551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:54.971349 (+ 798us) server_negotiation.cc:407] Connection header received
0929 02:05:54.971846 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:54.971851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:54.971924 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:54.972018 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:54.973846 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.975003 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.977656 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.978235 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.978430 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:54.981230 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:54.981235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:54.981238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:54.981299 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:54.984685 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:54.985992 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:54.986002 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:54.986009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:54.986199 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:54.986749 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:54.986757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:54.986761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:54.987148 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:54.987335 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:54.987769 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 02:05:54.988010 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:05:54.988533 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:54.970275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44461 (local address 127.11.182.193:41037)
0929 02:05:54.971191 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:54.971217 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:54.971520 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:54.972092 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:54.972099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:54.972992 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:54.973631 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:54.973651 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.975132 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.975140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:54.977441 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:54.977463 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:54.978587 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:54.978601 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:54.978784 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:54.979582 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:54.979603 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:54.980967 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:54.984925 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:54.984931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:54.984939 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:54.985755 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:54.986379 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:54.986387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:54.986394 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:54.986593 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:54.987285 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:54.987290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:54.987486 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 02:05:54.987648 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 02:05:54.988033 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":431,"threads_started":1}
I20260929 02:05:54.997357 13197 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:33545
I20260929 02:05:54.997359 13332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:33545 every 8 connection(s)
I20260929 02:05:54.998731 13197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:05:54.999374 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 13197
I20260929 02:05:54.999619 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:05:55.000217 11995 external_mini_cluster.cc:1428] Setting key 377f2367f012c104f3cce9374e2cfe84
I20260929 02:05:55.001518 13333 heartbeater.cc:347] Connected to a master server at 127.11.182.254:44461
I20260929 02:05:55.001842 13333 heartbeater.cc:464] Registering TS with master...
I20260929 02:05:55.002934 13333 heartbeater.cc:511] Master 127.11.182.254:44461 requested a full tablet report, sending...
I20260929 02:05:55.005621 13126 ts_manager.cc:194] Registered new tserver with Master: 348a62b8c67e465f93ba8a5500720d10 (127.11.182.193:33545)
I20260929 02:05:55.007615 13126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:46641
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:05:55.044761 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:44461
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:55.390928 13337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:55.391220 13337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:55.391320 13337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:55.398695 13337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:55.398824 13337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:55.398941 13337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:05:55.399024 13337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:05:55.408996 13337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.182.254:44461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:55.410246 13337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:55.411751 13337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:55.425793 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:55.425827 13345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:55.425850 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:55.427192 13337 server_base.cc:1034] running on GCE node
I20260929 02:05:55.427968 13337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:55.428973 13337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:55.430202 13337 hybrid_clock.cc:648] HybridClock initialized: now 1790647555430134 us; error 41 us; skew 500 ppm
I20260929 02:05:55.430552 13337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:55 dist-test-slave-q06p krb5kdc[13081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:55.434213 13337 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:05:55.435948 13337 webserver.cc:460] Webserver started at http://127.11.182.194:42831/ using document root <none> and password file <none>
I20260929 02:05:55.436764 13337 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:55.436921 13337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:55.437279 13337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:55.440662 13337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "6374faa171184b3994a5308d2eca338e"
format_stamp: "Formatted at 2026-09-29 02:05:55 on dist-test-slave-q06p"
server_key: "32de285e1849c1b0ed2e8462f5d433c0"
server_key_iv: "c03e1a2b68af1e73382a21fdec5df981"
server_key_version: "encryptionkey@0"
I20260929 02:05:55.441582 13337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "6374faa171184b3994a5308d2eca338e"
format_stamp: "Formatted at 2026-09-29 02:05:55 on dist-test-slave-q06p"
server_key: "32de285e1849c1b0ed2e8462f5d433c0"
server_key_iv: "c03e1a2b68af1e73382a21fdec5df981"
server_key_version: "encryptionkey@0"
I20260929 02:05:55.447641 13337 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260929 02:05:55.451313 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:55.452683 13337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:05:55.452899 13337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "6374faa171184b3994a5308d2eca338e"
format_stamp: "Formatted at 2026-09-29 02:05:55 on dist-test-slave-q06p"
server_key: "32de285e1849c1b0ed2e8462f5d433c0"
server_key_iv: "c03e1a2b68af1e73382a21fdec5df981"
server_key_version: "encryptionkey@0"
I20260929 02:05:55.453162 13337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:55.478003 13337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:55.482730 13337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:55.483101 13337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:55.485735 13337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:05:55.485870 13337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:55.486019 13337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:05:55.486088 13337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:05:55 dist-test-slave-q06p krb5kdc[13081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:55.514602 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:55.486575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:54565 (local address 127.11.182.254:44461)
0929 02:05:55.486778 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:55.486782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:55.488700 (+ 1918us) server_negotiation.cc:407] Connection header received
0929 02:05:55.491203 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:55.491212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:55.491311 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:55.491382 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:55.494826 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:55.495895 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:55.498404 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:55.499240 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:55.499428 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:55.504468 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:55.504478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:55.504486 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:55.504543 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:55.508395 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:55.509669 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:55.509674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:55.509680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:55.509804 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:55.510264 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:55.510269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:55.510273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:55.510564 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:55.510721 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:55.514068 (+ 3347us) server_negotiation.cc:299] Negotiation successful
0929 02:05:55.514244 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:05:55.513835 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:55.487654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44461 (local address 127.11.182.194:54565)
0929 02:05:55.488521 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:55.488569 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:55.490753 (+ 2184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:55.492058 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:55.492071 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:55.493066 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:55.494603 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:55.494641 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:55.496129 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:55.496135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:55.498151 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:55.498165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:55.499560 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:55.499570 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:55.499866 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:55.500832 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:55.500873 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:55.504183 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:55.508596 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:55.508603 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:55.508618 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:55.509474 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:55.509939 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:55.509948 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:55.509950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:55.510144 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:55.510698 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:55.510707 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:55.511175 (+ 468us) client_negotiation.cc:769] Sending connection context
0929 02:05:55.511580 (+ 405us) client_negotiation.cc:240] Negotiation successful
0929 02:05:55.512209 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":390,"threads_started":1}
I20260929 02:05:55.534956 13337 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:34807
I20260929 02:05:55.534974 13471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:34807 every 8 connection(s)
I20260929 02:05:55.536403 13337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:05:55.536408 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:55.521117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:35421 (local address 127.11.182.254:44461)
0929 02:05:55.521357 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:55.521368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:55.522016 (+ 648us) server_negotiation.cc:407] Connection header received
0929 02:05:55.522376 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:55.522383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:55.522485 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:55.522592 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:55.523939 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:55.524712 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:55.526034 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:55.526420 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:55.526558 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:55.529054 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:55.529059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:55.529062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:55.529102 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:55.533248 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:55.534226 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:55.534234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:55.534237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:55.534401 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:55.534773 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:55.534780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:55.534785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:55.535284 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:55.535442 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:55.535828 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 02:05:55.536061 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:05:55.536547 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:55.521099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44461 (local address 127.11.182.194:35421)
0929 02:05:55.521879 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:55.521904 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:55.522176 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:55.522760 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:55.522763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:55.523475 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:55.523784 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:55.523794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:55.524859 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:55.524869 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:55.525891 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:55.525902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:55.526608 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:55.526614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:55.526785 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:55.527638 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:55.527654 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:55.528774 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:55.533465 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:55.533473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:55.533479 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:55.534047 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:55.534520 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:55.534525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:55.534529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:55.534664 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:55.535402 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:55.535406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:55.535602 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 02:05:55.535746 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 02:05:55.536090 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":380,"threads_started":1}
I20260929 02:05:55.538621 13472 heartbeater.cc:347] Connected to a master server at 127.11.182.254:44461
I20260929 02:05:55.538952 13472 heartbeater.cc:464] Registering TS with master...
I20260929 02:05:55.539947 13472 heartbeater.cc:511] Master 127.11.182.254:44461 requested a full tablet report, sending...
I20260929 02:05:55.542040 13126 ts_manager.cc:194] Registered new tserver with Master: 6374faa171184b3994a5308d2eca338e (127.11.182.194:34807)
I20260929 02:05:55.542814 13126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:54565
I20260929 02:05:55.544804 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 13337
I20260929 02:05:55.545029 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:05:55.545441 11995 external_mini_cluster.cc:1428] Setting key 18f402743263eb9ac704ae48dffe19ea
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:05:55.591250 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:44461
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:55.932852 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:55.933189 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:55.933377 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:55.941100 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:55.941247 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:55.941419 13476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:05:55.941550 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:05:55.952551 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.182.254:44461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:55.954064 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:55.955667 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:55.970772 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:55.970686 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:55.970713 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:55.971822 13476 server_base.cc:1034] running on GCE node
I20260929 02:05:55.972659 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:55.973690 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:55.975083 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790647555974980 us; error 63 us; skew 500 ppm
I20260929 02:05:55.975487 13476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:55 dist-test-slave-q06p krb5kdc[13081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:55.986927 13476 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:05:55.989028 13476 webserver.cc:460] Webserver started at http://127.11.182.195:39167/ using document root <none> and password file <none>
I20260929 02:05:55.989882 13476 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:55.990024 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:55.990376 13476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:55.993682 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "13ea0af01ff244c7872b39a951e4bcc0"
format_stamp: "Formatted at 2026-09-29 02:05:55 on dist-test-slave-q06p"
server_key: "ae9639cf1f57b6eb17fb448e5d8751b1"
server_key_iv: "34b17fafec16974e451ce2401f314549"
server_key_version: "encryptionkey@0"
I20260929 02:05:55.994558 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "13ea0af01ff244c7872b39a951e4bcc0"
format_stamp: "Formatted at 2026-09-29 02:05:55 on dist-test-slave-q06p"
server_key: "ae9639cf1f57b6eb17fb448e5d8751b1"
server_key_iv: "34b17fafec16974e451ce2401f314549"
server_key_version: "encryptionkey@0"
I20260929 02:05:56.001279 13476 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 02:05:56.005687 13491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:56.007382 13476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:05:56.007570 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "13ea0af01ff244c7872b39a951e4bcc0"
format_stamp: "Formatted at 2026-09-29 02:05:55 on dist-test-slave-q06p"
server_key: "ae9639cf1f57b6eb17fb448e5d8751b1"
server_key_iv: "34b17fafec16974e451ce2401f314549"
server_key_version: "encryptionkey@0"
I20260929 02:05:56.007815 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:56.010955 13333 heartbeater.cc:503] Master 127.11.182.254:44461 was elected leader, sending a full tablet report...
I20260929 02:05:56.044443 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:56.049028 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:56.049315 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:56.052310 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:05:56.052419 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:56.052560 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:05:56.052641 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:05:56 dist-test-slave-q06p krb5kdc[13081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:56.078269 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.052980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:36059 (local address 127.11.182.254:44461)
0929 02:05:56.053760 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.053765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.054694 (+ 929us) server_negotiation.cc:407] Connection header received
0929 02:05:56.057060 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.057066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.057156 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.057342 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:56.059353 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.060761 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.063751 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.064418 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.064592 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.070120 (+ 5528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:56.070130 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:56.070138 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:56.070196 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:56.074004 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:56.075299 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:56.075308 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:56.075313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:56.075516 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:56.076132 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:56.076137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:56.076143 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:56.076541 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:56.076701 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.077621 (+ 920us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.077867 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":485,"threads_started":1}
I20260929 02:05:56.079734 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.053641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44461 (local address 127.11.182.195:36059)
0929 02:05:56.054506 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.054550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.056768 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.057580 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.057588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.058100 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:56.059090 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.059129 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.060971 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.060980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.063540 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.063553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.064736 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.064746 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.065072 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.066063 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:56.066107 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:56.069787 (+ 3680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:56.074200 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:56.074215 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:56.074244 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:56.075107 (+ 863us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:56.075640 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:56.075647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:56.075653 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:56.075990 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:56.076675 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:56.076684 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:56.077211 (+ 527us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.077502 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.078092 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":380,"threads_started":1}
I20260929 02:05:56.101480 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.085990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:54953 (local address 127.11.182.254:44461)
0929 02:05:56.086220 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.086228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.087062 (+ 834us) server_negotiation.cc:407] Connection header received
0929 02:05:56.087390 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.087398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.087496 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.087605 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:56.089364 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.090436 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.092512 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.092967 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.093127 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.095188 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:56.095195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:56.095199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:56.095270 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:56.098676 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:56.099498 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:56.099504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:56.099508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:56.099645 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:56.100015 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:56.100020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:56.100024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:56.100314 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:56.100475 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.100881 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.101106 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:05:56.101577 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.086027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44461 (local address 127.11.182.195:54953)
0929 02:05:56.086906 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.086933 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.087149 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.087767 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.087770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.088640 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:56.089191 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.089203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.090600 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.090606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.092346 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.092365 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.093178 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.093184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.093356 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.094109 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:56.094131 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:56.094983 (+ 852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:56.098839 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:56.098877 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:56.098882 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:56.099335 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:56.099762 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:56.099767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:56.099771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:56.099902 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:56.100429 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:56.100434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:56.100616 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.100752 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.101095 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":433,"threads_started":1}
I20260929 02:05:56.111161 13476 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:45197
I20260929 02:05:56.111182 13611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:45197 every 8 connection(s)
I20260929 02:05:56.112672 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:05:56.114993 13612 heartbeater.cc:347] Connected to a master server at 127.11.182.254:44461
I20260929 02:05:56.115201 13612 heartbeater.cc:464] Registering TS with master...
I20260929 02:05:56.115851 13612 heartbeater.cc:511] Master 127.11.182.254:44461 requested a full tablet report, sending...
I20260929 02:05:56.117316 13126 ts_manager.cc:194] Registered new tserver with Master: 13ea0af01ff244c7872b39a951e4bcc0 (127.11.182.195:45197)
I20260929 02:05:56.118247 13126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:36059
I20260929 02:05:56.122694 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 13476
I20260929 02:05:56.122953 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:05:56.123384 11995 external_mini_cluster.cc:1428] Setting key 84bc13e5357d9cc13dd16ea477ad7b9b
I20260929 02:05:56.126689 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:05:56 dist-test-slave-q06p krb5kdc[13081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:05:56 dist-test-slave-q06p krb5kdc[13081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:56.177850 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.161805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.11.182.254:44461)
0929 02:05:56.162080 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.162088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.162340 (+ 252us) server_negotiation.cc:407] Connection header received
0929 02:05:56.162577 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.162582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.162662 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.162754 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:56.164429 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.165299 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.167190 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.167636 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.167753 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.171992 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:56.172001 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:56.172007 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:56.172089 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:56.175142 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:56.175894 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:56.175899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:56.175901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:56.176038 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:56.176514 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:56.176518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:56.176520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:56.176817 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:56.177025 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.177433 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.177602 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:05:56.182814 13126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:05:56.189841 13126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:05:56.214602 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.203573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:34807 (local address 127.0.0.1:50534)
0929 02:05:56.204465 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.204507 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.204684 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.205930 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.205938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.205969 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:56.206354 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.206367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.208519 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.208526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.211163 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.211181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.213562 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.213569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.213795 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.213868 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.213985 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.214195 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":465,"threads_started":1}
I20260929 02:05:56.215222 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.203945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.11.182.194:34807)
0929 02:05:56.205163 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.205172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.205202 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:05:56.205325 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.205334 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.205647 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.206567 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:56.206802 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.208353 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.211426 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.213305 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.214393 (+ 1088us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.214605 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.214698 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.214922 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":906,"threads_started":1}
I20260929 02:05:56.215524 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.203581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45197 (local address 127.0.0.1:39758)
0929 02:05:56.205244 (+ 1663us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.205279 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.205388 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.206762 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.206767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.206793 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:56.207275 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.207289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.209836 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.209842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.212418 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.212436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.214455 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.214461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.214726 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.214817 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.214914 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.215056 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1520,"mutex_wait_us":529,"spinlock_wait_cycles":6912,"thread_start_us":951,"threads_started":1}
I20260929 02:05:56.216490 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.203879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.11.182.195:45197)
0929 02:05:56.206042 (+ 2163us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.206050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.206087 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:05:56.206189 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.206197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.206518 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.207567 (+ 1049us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:56.207960 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.209658 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.212678 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.214218 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.215635 (+ 1417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.215818 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.215935 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.216110 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1995,"thread_start_us":1799,"threads_started":1}
I20260929 02:05:56.219466 13402 tablet_service.cc:1467] Processing CreateTablet for tablet 9731485406eb47f392f6f6170b64e6e2 (DEFAULT_TABLE table=test-table [id=fff53eb780294482a2bcb549ed8767d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:05:56.221156 13402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9731485406eb47f392f6f6170b64e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:56.220357 13542 tablet_service.cc:1467] Processing CreateTablet for tablet 9731485406eb47f392f6f6170b64e6e2 (DEFAULT_TABLE table=test-table [id=fff53eb780294482a2bcb549ed8767d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:05:56.222124 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9731485406eb47f392f6f6170b64e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:56.224314 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.205789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:33545 (local address 127.0.0.1:41888)
0929 02:05:56.213473 (+ 7684us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.213497 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.213627 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.215146 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.215152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.215186 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:56.216353 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.216367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.219429 (+ 3062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.219437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.221449 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.221467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.223424 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.223433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.223628 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.223666 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.223788 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.223953 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7123,"thread_start_us":6997,"threads_started":1}
I20260929 02:05:56.224327 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.206396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.11.182.193:33545)
0929 02:05:56.214138 (+ 7742us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.214147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.214177 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:05:56.214292 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.214302 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.214667 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.214920 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:56.217348 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.219245 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.221668 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.223144 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.223349 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.223690 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.223837 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.224024 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7574,"thread_start_us":7580,"threads_started":1}
I20260929 02:05:56.229450 13263 tablet_service.cc:1467] Processing CreateTablet for tablet 9731485406eb47f392f6f6170b64e6e2 (DEFAULT_TABLE table=test-table [id=fff53eb780294482a2bcb549ed8767d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:05:56.231182 13263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9731485406eb47f392f6f6170b64e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:56.234870 13631 tablet_bootstrap.cc:492] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: Bootstrap starting.
I20260929 02:05:56.238742 13631 tablet_bootstrap.cc:654] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:56.240350 13631 log.cc:824] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:05:56.241477 13632 tablet_bootstrap.cc:492] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: Bootstrap starting.
I20260929 02:05:56.245555 13632 tablet_bootstrap.cc:654] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:56.245632 13631 tablet_bootstrap.cc:492] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: No bootstrap required, opened a new log
I20260929 02:05:56.245922 13631 ts_tablet_manager.cc:1397] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 02:05:56.247038 13632 log.cc:824] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:05:56.249248 13635 tablet_bootstrap.cc:492] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: Bootstrap starting.
I20260929 02:05:56.250965 13632 tablet_bootstrap.cc:492] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: No bootstrap required, opened a new log
I20260929 02:05:56.251271 13632 ts_tablet_manager.cc:1397] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 02:05:56.252974 13635 tablet_bootstrap.cc:654] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:56.254931 13635 log.cc:824] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: Log is configured to *not* fsync() on all Append() calls
I20260929 02:05:56.258247 13635 tablet_bootstrap.cc:492] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: No bootstrap required, opened a new log
I20260929 02:05:56.258603 13635 ts_tablet_manager.cc:1397] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260929 02:05:56.258397 13631 raft_consensus.cc:348] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } }
I20260929 02:05:56.258909 13631 raft_consensus.cc:374] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:56.259001 13631 raft_consensus.cc:729] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13ea0af01ff244c7872b39a951e4bcc0, State: Initialized, Role: FOLLOWER
I20260929 02:05:56.259716 13631 consensus_queue.cc:260] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } }
I20260929 02:05:56.260921 13612 heartbeater.cc:503] Master 127.11.182.254:44461 was elected leader, sending a full tablet report...
I20260929 02:05:56.263264 13632 raft_consensus.cc:348] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } }
I20260929 02:05:56.263665 13631 ts_tablet_manager.cc:1428] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260929 02:05:56.263811 13632 raft_consensus.cc:374] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:56.263928 13632 raft_consensus.cc:729] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6374faa171184b3994a5308d2eca338e, State: Initialized, Role: FOLLOWER
I20260929 02:05:56.264938 13632 consensus_queue.cc:260] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } }
I20260929 02:05:56.272217 13472 heartbeater.cc:503] Master 127.11.182.254:44461 was elected leader, sending a full tablet report...
I20260929 02:05:56.273195 13632 ts_tablet_manager.cc:1428] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: Time spent starting tablet: real 0.022s user 0.018s sys 0.004s
I20260929 02:05:56.273324 13635 raft_consensus.cc:348] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } }
I20260929 02:05:56.273811 13635 raft_consensus.cc:374] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:56.273922 13635 raft_consensus.cc:729] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348a62b8c67e465f93ba8a5500720d10, State: Initialized, Role: FOLLOWER
I20260929 02:05:56.274536 13635 consensus_queue.cc:260] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } }
I20260929 02:05:56.276912 13635 ts_tablet_manager.cc:1428] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
W20260929 02:05:56.290359 13473 tablet.cc:2369] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:05:56.364625 13613 tablet.cc:2369] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:05:56.438354 13639 raft_consensus.cc:482] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:05:56.438684 13639 raft_consensus.cc:504] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } }
I20260929 02:05:56.440487 13639 leader_election.cc:288] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13ea0af01ff244c7872b39a951e4bcc0 (127.11.182.195:45197), 6374faa171184b3994a5308d2eca338e (127.11.182.194:34807)
I20260929 02:05:56.447700 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.440921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:48937 (local address 127.11.182.194:34807)
0929 02:05:56.441118 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.441125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.442351 (+ 1226us) server_negotiation.cc:407] Connection header received
0929 02:05:56.442618 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.442622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.442683 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.442784 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:56.443299 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.444044 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.445980 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.446737 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.446956 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.447032 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.447309 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.447434 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:05:56.447858 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.441018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:34807 (local address 127.11.182.193:48937)
0929 02:05:56.442225 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.442241 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.442446 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.442798 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.442803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.442827 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:56.443157 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.443166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.444183 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.444191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.445808 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.445820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.446969 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.446975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.447143 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.447166 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.447271 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.447408 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":61,"thread_start_us":500,"threads_started":1}
I20260929 02:05:56.447857 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.440930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:60607 (local address 127.11.182.195:45197)
0929 02:05:56.441118 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.441125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.442720 (+ 1595us) server_negotiation.cc:407] Connection header received
0929 02:05:56.442941 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.442948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.443010 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.443104 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:56.443662 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.444396 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.446123 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.446949 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.447121 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.447195 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.447481 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.447604 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:05:56.448014 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.441047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45197 (local address 127.11.182.193:60607)
0929 02:05:56.442602 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.442620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.442771 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.443134 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.443144 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.443169 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:56.443505 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.443513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.444529 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.444534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.445969 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.445980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.447161 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.447168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.447321 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.447343 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.447459 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.447585 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"mutex_wait_us":1,"thread_start_us":1244,"threads_started":1}
I20260929 02:05:56.448907 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9731485406eb47f392f6f6170b64e6e2" candidate_uuid: "348a62b8c67e465f93ba8a5500720d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6374faa171184b3994a5308d2eca338e" is_pre_election: true
I20260929 02:05:56.448907 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9731485406eb47f392f6f6170b64e6e2" candidate_uuid: "348a62b8c67e465f93ba8a5500720d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13ea0af01ff244c7872b39a951e4bcc0" is_pre_election: true
I20260929 02:05:56.449399 13567 raft_consensus.cc:2454] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 348a62b8c67e465f93ba8a5500720d10 in term 0.
I20260929 02:05:56.449405 13422 raft_consensus.cc:2454] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 348a62b8c67e465f93ba8a5500720d10 in term 0.
I20260929 02:05:56.450539 13213 leader_election.cc:302] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348a62b8c67e465f93ba8a5500720d10, 6374faa171184b3994a5308d2eca338e; no voters:
I20260929 02:05:56.451083 13639 raft_consensus.cc:2781] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:05:56.451238 13639 raft_consensus.cc:482] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:05:56.451364 13639 raft_consensus.cc:3037] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:56.453632 13639 raft_consensus.cc:504] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } }
I20260929 02:05:56.454349 13639 leader_election.cc:288] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [CANDIDATE]: Term 1 election: Requested vote from peers 13ea0af01ff244c7872b39a951e4bcc0 (127.11.182.195:45197), 6374faa171184b3994a5308d2eca338e (127.11.182.194:34807)
I20260929 02:05:56.454828 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9731485406eb47f392f6f6170b64e6e2" candidate_uuid: "348a62b8c67e465f93ba8a5500720d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13ea0af01ff244c7872b39a951e4bcc0"
I20260929 02:05:56.454921 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9731485406eb47f392f6f6170b64e6e2" candidate_uuid: "348a62b8c67e465f93ba8a5500720d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6374faa171184b3994a5308d2eca338e"
I20260929 02:05:56.455109 13567 raft_consensus.cc:3037] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:56.455112 13422 raft_consensus.cc:3037] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:56.458482 13422 raft_consensus.cc:2454] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 348a62b8c67e465f93ba8a5500720d10 in term 1.
I20260929 02:05:56.458482 13567 raft_consensus.cc:2454] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 348a62b8c67e465f93ba8a5500720d10 in term 1.
I20260929 02:05:56.459050 13213 leader_election.cc:302] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13ea0af01ff244c7872b39a951e4bcc0, 348a62b8c67e465f93ba8a5500720d10, 6374faa171184b3994a5308d2eca338e; no voters:
I20260929 02:05:56.459544 13639 raft_consensus.cc:2781] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:05:56.459949 13639 raft_consensus.cc:686] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 1 LEADER]: Becoming Leader. State: Replica: 348a62b8c67e465f93ba8a5500720d10, State: Running, Role: LEADER
I20260929 02:05:56.460487 13639 consensus_queue.cc:237] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } }
I20260929 02:05:56.465586 13126 catalog_manager.cc:5495] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 348a62b8c67e465f93ba8a5500720d10 (127.11.182.193). New cstate: current_term: 1 leader_uuid: "348a62b8c67e465f93ba8a5500720d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } health_report { overall_health: HEALTHY } } }
W20260929 02:05:56.507376 13334 tablet.cc:2369] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:05:56.535024 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.528218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.11.182.193:33545)
0929 02:05:56.528455 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.528459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.528664 (+ 205us) server_negotiation.cc:407] Connection header received
0929 02:05:56.529164 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.529169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.529267 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.529411 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:05:56.529845 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.530570 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.532157 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.532625 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.532740 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.533460 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:05:56.534252 (+ 792us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:05:56.534423 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.534609 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.534696 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 02:05:56.544610 13422 raft_consensus.cc:1261] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [term 1 FOLLOWER]: Refusing update from remote peer 348a62b8c67e465f93ba8a5500720d10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:05:56.544608 13567 raft_consensus.cc:1261] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [term 1 FOLLOWER]: Refusing update from remote peer 348a62b8c67e465f93ba8a5500720d10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:05:56.545811 13642 consensus_queue.cc:1035] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:05:56.546177 13639 consensus_queue.cc:1035] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:05:56.582983 13126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:05:56.583302 13126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:05:56.587227 13126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90: Sending DeleteTablet for 3 replicas of tablet 9731485406eb47f392f6f6170b64e6e2
I20260929 02:05:56.588531 13402 tablet_service.cc:1514] Processing DeleteTablet for tablet 9731485406eb47f392f6f6170b64e6e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:05:56 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:50534
I20260929 02:05:56.588585 13263 tablet_service.cc:1514] Processing DeleteTablet for tablet 9731485406eb47f392f6f6170b64e6e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:05:56 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:41888
I20260929 02:05:56.588585 13542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9731485406eb47f392f6f6170b64e6e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:05:56 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:39758
I20260929 02:05:56.590385 13125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:05:56.591555 13125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:05:56.595877 13658 tablet_replica.cc:330] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: stopping tablet replica
I20260929 02:05:56.596460 13658 raft_consensus.cc:2229] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:05:56.597016 13658 raft_consensus.cc:2258] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:05:56.598026 13657 tablet_replica.cc:330] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: stopping tablet replica
I20260929 02:05:56.598520 13657 raft_consensus.cc:2229] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:05:56.599570 13657 raft_consensus.cc:2258] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:05:56.601651 13658 ts_tablet_manager.cc:1905] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:05:56.603268 13402 tablet_service.cc:1467] Processing CreateTablet for tablet b42a46f2528645e991bfa293c7c9ade6 (DEFAULT_TABLE table=test-table [id=92ea36529a034db297eb957e16970c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:05:56.603377 13263 tablet_service.cc:1467] Processing CreateTablet for tablet b42a46f2528645e991bfa293c7c9ade6 (DEFAULT_TABLE table=test-table [id=92ea36529a034db297eb957e16970c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:05:56.603762 13402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b42a46f2528645e991bfa293c7c9ade6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:56.603852 13263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b42a46f2528645e991bfa293c7c9ade6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:56.599341 13656 tablet_replica.cc:330] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: stopping tablet replica
I20260929 02:05:56.609424 13656 raft_consensus.cc:2229] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:05:56.609913 13656 raft_consensus.cc:2258] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:05:56.610435 13658 ts_tablet_manager.cc:1918] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:05:56.610625 13658 log.cc:1196] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/9731485406eb47f392f6f6170b64e6e2
I20260929 02:05:56.611405 13658 ts_tablet_manager.cc:1939] T 9731485406eb47f392f6f6170b64e6e2 P 6374faa171184b3994a5308d2eca338e: Deleting consensus metadata
I20260929 02:05:56.612587 13656 ts_tablet_manager.cc:1905] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:05:56.613514 13110 catalog_manager.cc:4841] TS 6374faa171184b3994a5308d2eca338e (127.11.182.194:34807): tablet 9731485406eb47f392f6f6170b64e6e2 (table test-table [id=fff53eb780294482a2bcb549ed8767d5]) successfully deleted
I20260929 02:05:56.614979 13632 tablet_bootstrap.cc:492] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: Bootstrap starting.
I20260929 02:05:56.616146 13542 tablet_service.cc:1467] Processing CreateTablet for tablet b42a46f2528645e991bfa293c7c9ade6 (DEFAULT_TABLE table=test-table [id=92ea36529a034db297eb957e16970c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:05:56.616706 13632 tablet_bootstrap.cc:654] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:56.617115 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b42a46f2528645e991bfa293c7c9ade6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:56.618736 13632 tablet_bootstrap.cc:492] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: No bootstrap required, opened a new log
I20260929 02:05:56.618947 13632 ts_tablet_manager.cc:1397] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 02:05:56.619684 13632 raft_consensus.cc:348] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.619870 13635 tablet_bootstrap.cc:492] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: Bootstrap starting.
I20260929 02:05:56.619928 13632 raft_consensus.cc:374] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:56.620012 13632 raft_consensus.cc:729] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6374faa171184b3994a5308d2eca338e, State: Initialized, Role: FOLLOWER
I20260929 02:05:56.620298 13632 consensus_queue.cc:260] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.620972 13632 ts_tablet_manager.cc:1428] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:05:56.621526 13635 tablet_bootstrap.cc:654] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:56.625622 13656 ts_tablet_manager.cc:1918] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:05:56.625792 13635 tablet_bootstrap.cc:492] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: No bootstrap required, opened a new log
I20260929 02:05:56.625836 13656 log.cc:1196] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/9731485406eb47f392f6f6170b64e6e2
I20260929 02:05:56.625962 13635 ts_tablet_manager.cc:1397] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260929 02:05:56.626384 13656 ts_tablet_manager.cc:1939] T 9731485406eb47f392f6f6170b64e6e2 P 13ea0af01ff244c7872b39a951e4bcc0: Deleting consensus metadata
I20260929 02:05:56.626745 13635 raft_consensus.cc:348] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.627025 13635 raft_consensus.cc:374] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:56.627152 13635 raft_consensus.cc:729] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348a62b8c67e465f93ba8a5500720d10, State: Initialized, Role: FOLLOWER
I20260929 02:05:56.627496 13635 consensus_queue.cc:260] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.628770 13112 catalog_manager.cc:4841] TS 13ea0af01ff244c7872b39a951e4bcc0 (127.11.182.195:45197): tablet 9731485406eb47f392f6f6170b64e6e2 (table test-table [id=fff53eb780294482a2bcb549ed8767d5]) successfully deleted
I20260929 02:05:56.630074 13657 ts_tablet_manager.cc:1905] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:05:56.631503 13635 ts_tablet_manager.cc:1428] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:05:56.634197 13631 tablet_bootstrap.cc:492] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: Bootstrap starting.
I20260929 02:05:56.636089 13631 tablet_bootstrap.cc:654] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:56.638948 13631 tablet_bootstrap.cc:492] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: No bootstrap required, opened a new log
I20260929 02:05:56.639123 13631 ts_tablet_manager.cc:1397] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 02:05:56.639429 13657 ts_tablet_manager.cc:1918] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:05:56.639652 13657 log.cc:1196] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/9731485406eb47f392f6f6170b64e6e2
I20260929 02:05:56.639916 13631 raft_consensus.cc:348] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.640122 13631 raft_consensus.cc:374] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:56.640213 13657 ts_tablet_manager.cc:1939] T 9731485406eb47f392f6f6170b64e6e2 P 348a62b8c67e465f93ba8a5500720d10: Deleting consensus metadata
I20260929 02:05:56.640215 13631 raft_consensus.cc:729] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13ea0af01ff244c7872b39a951e4bcc0, State: Initialized, Role: FOLLOWER
I20260929 02:05:56.640525 13631 consensus_queue.cc:260] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.641194 13631 ts_tablet_manager.cc:1428] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:05:56.642140 13112 catalog_manager.cc:4841] TS 348a62b8c67e465f93ba8a5500720d10 (127.11.182.193:33545): tablet 9731485406eb47f392f6f6170b64e6e2 (table test-table [id=fff53eb780294482a2bcb549ed8767d5]) successfully deleted
I20260929 02:05:56.749122 13638 raft_consensus.cc:482] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:05:56.749398 13638 raft_consensus.cc:504] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.751255 13638 leader_election.cc:288] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 348a62b8c67e465f93ba8a5500720d10 (127.11.182.193:33545), 13ea0af01ff244c7872b39a951e4bcc0 (127.11.182.195:45197)
I20260929 02:05:56.759742 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.751675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:51525 (local address 127.11.182.193:33545)
0929 02:05:56.751855 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.751860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.752359 (+ 499us) server_negotiation.cc:407] Connection header received
0929 02:05:56.752617 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.752623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.752707 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.752805 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:56.753663 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.754933 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.757165 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.758278 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.758461 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.758551 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.759252 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.759401 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:05:56.759793 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.751695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:33545 (local address 127.11.182.194:51525)
0929 02:05:56.752237 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.752253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.752430 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.753046 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.753053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.753096 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:56.753507 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.753519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.755064 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.755070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.757003 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.757016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.758778 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.758800 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.759084 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.759112 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.759225 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.759381 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":28,"thread_start_us":294,"threads_started":1}
I20260929 02:05:56.761022 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.752657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:34969 (local address 127.11.182.195:45197)
0929 02:05:56.752830 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.752837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.753439 (+ 602us) server_negotiation.cc:407] Connection header received
0929 02:05:56.753709 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.753714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.753773 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.753853 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:05:56.754547 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.755784 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.758456 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.759747 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.759883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.759963 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.760593 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.760746 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:05:56.761334 13288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b42a46f2528645e991bfa293c7c9ade6" candidate_uuid: "6374faa171184b3994a5308d2eca338e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348a62b8c67e465f93ba8a5500720d10" is_pre_election: true
I20260929 02:05:56.761781 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.752512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45197 (local address 127.11.182.194:34969)
0929 02:05:56.753312 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.753337 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.753506 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.754038 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.754041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.754062 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:05:56.754318 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.754330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.755983 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.755990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.758222 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.758238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.760166 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.760172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.760388 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.760423 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.761193 (+ 770us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.761393 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":1,"thread_start_us":452,"threads_started":1}
I20260929 02:05:56.761912 13288 raft_consensus.cc:2454] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6374faa171184b3994a5308d2eca338e in term 0.
I20260929 02:05:56.762439 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b42a46f2528645e991bfa293c7c9ade6" candidate_uuid: "6374faa171184b3994a5308d2eca338e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13ea0af01ff244c7872b39a951e4bcc0" is_pre_election: true
I20260929 02:05:56.762737 13567 raft_consensus.cc:2454] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6374faa171184b3994a5308d2eca338e in term 0.
I20260929 02:05:56.762641 13355 leader_election.cc:302] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348a62b8c67e465f93ba8a5500720d10, 6374faa171184b3994a5308d2eca338e; no voters:
I20260929 02:05:56.763099 13638 raft_consensus.cc:2781] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:05:56.763235 13638 raft_consensus.cc:482] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:05:56.763355 13638 raft_consensus.cc:3037] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:56.765974 13638 raft_consensus.cc:504] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.766577 13638 leader_election.cc:288] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [CANDIDATE]: Term 1 election: Requested vote from peers 348a62b8c67e465f93ba8a5500720d10 (127.11.182.193:33545), 13ea0af01ff244c7872b39a951e4bcc0 (127.11.182.195:45197)
I20260929 02:05:56.767239 13288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b42a46f2528645e991bfa293c7c9ade6" candidate_uuid: "6374faa171184b3994a5308d2eca338e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348a62b8c67e465f93ba8a5500720d10"
I20260929 02:05:56.767863 13288 raft_consensus.cc:3037] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:56.770144 13288 raft_consensus.cc:2454] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6374faa171184b3994a5308d2eca338e in term 1.
I20260929 02:05:56.770566 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b42a46f2528645e991bfa293c7c9ade6" candidate_uuid: "6374faa171184b3994a5308d2eca338e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13ea0af01ff244c7872b39a951e4bcc0"
I20260929 02:05:56.770658 13355 leader_election.cc:302] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348a62b8c67e465f93ba8a5500720d10, 6374faa171184b3994a5308d2eca338e; no voters:
I20260929 02:05:56.770809 13567 raft_consensus.cc:3037] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:56.771010 13638 raft_consensus.cc:2781] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:05:56.771936 13638 raft_consensus.cc:686] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 1 LEADER]: Becoming Leader. State: Replica: 6374faa171184b3994a5308d2eca338e, State: Running, Role: LEADER
I20260929 02:05:56.772419 13638 consensus_queue.cc:237] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.774036 13567 raft_consensus.cc:2454] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6374faa171184b3994a5308d2eca338e in term 1.
I20260929 02:05:56.776880 13126 catalog_manager.cc:5495] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6374faa171184b3994a5308d2eca338e (127.11.182.194). New cstate: current_term: 1 leader_uuid: "6374faa171184b3994a5308d2eca338e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:05:56 dist-test-slave-q06p krb5kdc[13081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:56.858099 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.841611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44461 (local address 127.0.0.1:53168)
0929 02:05:56.842874 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:56.842901 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:56.843080 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:56.843461 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:56.843465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:56.844247 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:05:56.844460 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.844470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.845634 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.845640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.846662 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:56.846673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.847591 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.847598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:56.847804 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.848576 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:05:56.848595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:05:56.851241 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:05:56.854920 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:56.854940 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:56.854949 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:56.855459 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:56.855881 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:05:56.855887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:05:56.855891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:05:56.856054 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:05:56.856750 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:05:56.856761 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:05:56.856978 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 02:05:56.857200 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 02:05:56.857563 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":425,"threads_started":1}
I20260929 02:05:56.858433 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:56.841661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.11.182.254:44461)
0929 02:05:56.842992 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:56.842998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:56.843026 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:05:56.843241 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:56.843248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:56.843310 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:56.843436 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:56.844622 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.845466 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.846821 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:56.847326 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:56.847572 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:56.851474 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:56.851480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:56.851484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:56.851516 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:56.854720 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:56.855596 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:56.855604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:56.855608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:56.855756 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:56.856176 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:56.856189 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:56.856192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:56.856557 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:56.856833 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:56.857295 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 02:05:56.857682 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":1190,"thread_start_us":978,"threads_started":1}
I20260929 02:05:56.865062 13126 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:53168:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:05:56.866406 13126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:05:56.874326 13263 tablet_service.cc:1467] Processing CreateTablet for tablet d8f2c2eda755484789ad350375000e78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c1c0a323aea449ea9368fde385e511a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:05:56.874673 13402 tablet_service.cc:1467] Processing CreateTablet for tablet d8f2c2eda755484789ad350375000e78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c1c0a323aea449ea9368fde385e511a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:05:56.875094 13263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f2c2eda755484789ad350375000e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:56.875239 13402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f2c2eda755484789ad350375000e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:56.875061 13542 tablet_service.cc:1467] Processing CreateTablet for tablet d8f2c2eda755484789ad350375000e78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c1c0a323aea449ea9368fde385e511a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:05:56.875608 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f2c2eda755484789ad350375000e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:56.883795 13631 tablet_bootstrap.cc:492] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: Bootstrap starting.
I20260929 02:05:56.885663 13631 tablet_bootstrap.cc:654] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:56.886772 13635 tablet_bootstrap.cc:492] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: Bootstrap starting.
I20260929 02:05:56.888571 13635 tablet_bootstrap.cc:654] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:56.889925 13632 tablet_bootstrap.cc:492] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: Bootstrap starting.
I20260929 02:05:56.891549 13635 tablet_bootstrap.cc:492] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: No bootstrap required, opened a new log
I20260929 02:05:56.892202 13632 tablet_bootstrap.cc:654] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:56.892184 13635 ts_tablet_manager.cc:1397] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 02:05:56.893057 13635 raft_consensus.cc:348] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.893373 13635 raft_consensus.cc:374] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:56.893483 13635 raft_consensus.cc:729] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348a62b8c67e465f93ba8a5500720d10, State: Initialized, Role: FOLLOWER
I20260929 02:05:56.893806 13631 tablet_bootstrap.cc:492] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: No bootstrap required, opened a new log
I20260929 02:05:56.894002 13631 ts_tablet_manager.cc:1397] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260929 02:05:56.893823 13635 consensus_queue.cc:260] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.894544 13635 ts_tablet_manager.cc:1428] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:05:56.894405 13654 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:56.894788 13654 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:05:56.895650 13631 raft_consensus.cc:348] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.895921 13631 raft_consensus.cc:374] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:56.896001 13631 raft_consensus.cc:729] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13ea0af01ff244c7872b39a951e4bcc0, State: Initialized, Role: FOLLOWER
I20260929 02:05:56.896651 13631 consensus_queue.cc:260] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.897292 13631 ts_tablet_manager.cc:1428] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:05:56.897200 13637 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:56.897568 13637 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:05:56.899061 13632 tablet_bootstrap.cc:492] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: No bootstrap required, opened a new log
I20260929 02:05:56.899253 13632 ts_tablet_manager.cc:1397] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 02:05:56.900128 13632 raft_consensus.cc:348] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.900707 13632 raft_consensus.cc:374] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:56.900811 13632 raft_consensus.cc:729] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6374faa171184b3994a5308d2eca338e, State: Initialized, Role: FOLLOWER
I20260929 02:05:56.901085 13632 consensus_queue.cc:260] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:56.901695 13632 ts_tablet_manager.cc:1428] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:05:56.901554 13638 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:56.901844 13638 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:05:57.101884 13638 raft_consensus.cc:482] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:05:57.102146 13638 raft_consensus.cc:504] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:57.102759 13638 leader_election.cc:288] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 348a62b8c67e465f93ba8a5500720d10 (127.11.182.193:33545), 13ea0af01ff244c7872b39a951e4bcc0 (127.11.182.195:45197)
I20260929 02:05:57.103438 13288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f2c2eda755484789ad350375000e78" candidate_uuid: "6374faa171184b3994a5308d2eca338e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348a62b8c67e465f93ba8a5500720d10" is_pre_election: true
I20260929 02:05:57.103490 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f2c2eda755484789ad350375000e78" candidate_uuid: "6374faa171184b3994a5308d2eca338e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13ea0af01ff244c7872b39a951e4bcc0" is_pre_election: true
I20260929 02:05:57.103691 13288 raft_consensus.cc:2454] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6374faa171184b3994a5308d2eca338e in term 0.
I20260929 02:05:57.103820 13567 raft_consensus.cc:2454] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6374faa171184b3994a5308d2eca338e in term 0.
I20260929 02:05:57.104141 13355 leader_election.cc:302] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348a62b8c67e465f93ba8a5500720d10, 6374faa171184b3994a5308d2eca338e; no voters:
I20260929 02:05:57.104418 13638 raft_consensus.cc:2781] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:05:57.104519 13638 raft_consensus.cc:482] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:05:57.104653 13638 raft_consensus.cc:3037] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:57.106601 13638 raft_consensus.cc:504] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:57.107270 13638 leader_election.cc:288] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [CANDIDATE]: Term 1 election: Requested vote from peers 348a62b8c67e465f93ba8a5500720d10 (127.11.182.193:33545), 13ea0af01ff244c7872b39a951e4bcc0 (127.11.182.195:45197)
I20260929 02:05:57.107766 13288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f2c2eda755484789ad350375000e78" candidate_uuid: "6374faa171184b3994a5308d2eca338e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348a62b8c67e465f93ba8a5500720d10"
I20260929 02:05:57.107808 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f2c2eda755484789ad350375000e78" candidate_uuid: "6374faa171184b3994a5308d2eca338e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13ea0af01ff244c7872b39a951e4bcc0"
I20260929 02:05:57.108009 13288 raft_consensus.cc:3037] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:57.108042 13567 raft_consensus.cc:3037] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:57.109936 13567 raft_consensus.cc:2454] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6374faa171184b3994a5308d2eca338e in term 1.
I20260929 02:05:57.109993 13288 raft_consensus.cc:2454] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6374faa171184b3994a5308d2eca338e in term 1.
I20260929 02:05:57.110531 13355 leader_election.cc:302] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348a62b8c67e465f93ba8a5500720d10, 6374faa171184b3994a5308d2eca338e; no voters:
I20260929 02:05:57.110881 13638 raft_consensus.cc:2781] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:05:57.111090 13638 raft_consensus.cc:686] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [term 1 LEADER]: Becoming Leader. State: Replica: 6374faa171184b3994a5308d2eca338e, State: Running, Role: LEADER
I20260929 02:05:57.111361 13638 consensus_queue.cc:237] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } }
I20260929 02:05:57.113101 13664 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: TxnStatusTablet state changed. Reason: New leader 6374faa171184b3994a5308d2eca338e. Latest consensus state: current_term: 1 leader_uuid: "6374faa171184b3994a5308d2eca338e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:57.113396 13664 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:05:57.115448 13125 catalog_manager.cc:5495] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6374faa171184b3994a5308d2eca338e (127.11.182.194). New cstate: current_term: 1 leader_uuid: "6374faa171184b3994a5308d2eca338e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } health_report { overall_health: UNKNOWN } } }
I20260929 02:05:57.122088 13677 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:05:57.122359 13677 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:05:57.141458 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.132419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:34807 (local address 127.0.0.1:50544)
0929 02:05:57.132679 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:57.132700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:57.132834 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:57.134106 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:57.134114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:57.134136 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:05:57.134409 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:57.134421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.136051 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.136059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:57.138358 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:57.138381 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.139484 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.139490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:57.139700 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.139786 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:05:57.140880 (+ 1094us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:05:57.140909 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:05:57.141016 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 02:05:57.141173 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 02:05:57.141993 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.132993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.11.182.194:34807)
0929 02:05:57.133626 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:57.133632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:57.133651 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:05:57.133745 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:57.133749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:57.133868 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:57.134040 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:05:57.134578 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.135893 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.138555 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.139293 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.139467 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.139992 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:05:57.140741 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:05:57.140905 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:57.141074 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 02:05:57.141170 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":337,"threads_started":1}
I20260929 02:05:57.146476 13567 raft_consensus.cc:1261] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0 [term 1 FOLLOWER]: Refusing update from remote peer 6374faa171184b3994a5308d2eca338e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:05:57.146697 13288 raft_consensus.cc:1261] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10 [term 1 FOLLOWER]: Refusing update from remote peer 6374faa171184b3994a5308d2eca338e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:05:57.147311 13638 consensus_queue.cc:1035] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [LEADER]: Connected to new peer: Peer: permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:05:57.147657 13664 consensus_queue.cc:1035] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e [LEADER]: Connected to new peer: Peer: permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:05:57.152961 13637 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: TxnStatusTablet state changed. Reason: New leader 6374faa171184b3994a5308d2eca338e. Latest consensus state: current_term: 1 leader_uuid: "6374faa171184b3994a5308d2eca338e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:57.153227 13637 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:05:57.159162 13681 mvcc.cc:205] Tried to move back new op lower bound from 7334492394064502784 to 7334492393930776576. Current Snapshot: MvccSnapshot[applied={T|T < 7334492394064502784}]
I20260929 02:05:57.165892 13637 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6374faa171184b3994a5308d2eca338e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:57.166293 13637 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 13ea0af01ff244c7872b39a951e4bcc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:05:57.170794 13638 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6374faa171184b3994a5308d2eca338e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:57.171165 13638 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:05:57.172048 13654 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: TxnStatusTablet state changed. Reason: New leader 6374faa171184b3994a5308d2eca338e. Latest consensus state: current_term: 1 leader_uuid: "6374faa171184b3994a5308d2eca338e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:57.172324 13654 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:05:57.174608 13638 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6374faa171184b3994a5308d2eca338e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:57.174891 13638 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 6374faa171184b3994a5308d2eca338e: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:05:57.176534 13654 tablet_replica.cc:439] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6374faa171184b3994a5308d2eca338e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 } } peers { permanent_uuid: "6374faa171184b3994a5308d2eca338e" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34807 } } peers { permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 } } }
I20260929 02:05:57.176779 13654 tablet_replica.cc:442] T d8f2c2eda755484789ad350375000e78 P 348a62b8c67e465f93ba8a5500720d10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:05:57.185104 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.179411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.11.182.194:34807)
0929 02:05:57.179604 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:57.179613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:57.180117 (+ 504us) server_negotiation.cc:407] Connection header received
0929 02:05:57.180329 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:57.180334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:57.180420 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:57.180527 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:05:57.181024 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.181788 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.183423 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.183840 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.183983 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.184401 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:05:57.184501 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:05:57.184602 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:57.184724 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 02:05:57.184802 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:05:57.196261 13664 consensus_queue.cc:1035] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [LEADER]: Connected to new peer: Peer: permanent_uuid: "13ea0af01ff244c7872b39a951e4bcc0" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:05:57.204378 13664 consensus_queue.cc:1035] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [LEADER]: Connected to new peer: Peer: permanent_uuid: "348a62b8c67e465f93ba8a5500720d10" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:05:57.208524 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.197081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:47175 (local address 127.11.182.194:34807)
0929 02:05:57.197305 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:57.197314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:57.198018 (+ 704us) server_negotiation.cc:407] Connection header received
0929 02:05:57.198921 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:57.198926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:57.199035 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:57.199097 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:05:57.199723 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.200681 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.206135 (+ 5454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.206916 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.207016 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.207663 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:05:57.207835 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:05:57.207900 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:57.208095 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 02:05:57.208183 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:05:57.209321 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.197151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:34807 (local address 127.11.182.194:47175)
0929 02:05:57.197906 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 02:05:57.197922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:05:57.198723 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:05:57.199269 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:05:57.199272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:05:57.199281 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:05:57.199581 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:57.199590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.200819 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.200825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:57.203299 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:05:57.203316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.207220 (+ 3904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.207228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:05:57.207400 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.207474 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:05:57.207981 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:05:57.207989 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:05:57.208750 (+ 761us) client_negotiation.cc:240] Negotiation successful
0929 02:05:57.208898 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":392,"threads_started":1}
I20260929 02:05:57.284704 13125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:05:57.285005 13125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:05:57.288750 13125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe2ea2ec89d14a5fb0259ff2c4d52c90: Sending DeleteTablet for 3 replicas of tablet b42a46f2528645e991bfa293c7c9ade6
I20260929 02:05:57.289731 13542 tablet_service.cc:1514] Processing DeleteTablet for tablet b42a46f2528645e991bfa293c7c9ade6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:05:57 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:39758
I20260929 02:05:57.289994 13263 tablet_service.cc:1514] Processing DeleteTablet for tablet b42a46f2528645e991bfa293c7c9ade6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:05:57 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:41888
I20260929 02:05:57.290179 13401 tablet_service.cc:1514] Processing DeleteTablet for tablet b42a46f2528645e991bfa293c7c9ade6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:05:57 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:50534
I20260929 02:05:57.297806 13701 tablet_replica.cc:330] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: stopping tablet replica
I20260929 02:05:57.298159 13701 raft_consensus.cc:2229] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:05:57.301239 13701 raft_consensus.cc:2258] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:05:57.303023 13700 tablet_replica.cc:330] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: stopping tablet replica
I20260929 02:05:57.303421 13700 raft_consensus.cc:2229] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:05:57.303732 13700 raft_consensus.cc:2258] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:05:57.304014 13701 ts_tablet_manager.cc:1905] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:05:57.304970 13700 ts_tablet_manager.cc:1905] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:05:57.306360 13699 tablet_replica.cc:330] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: stopping tablet replica
I20260929 02:05:57.306579 13699 raft_consensus.cc:2229] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:05:57.306800 13699 raft_consensus.cc:2258] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:05:57.308792 13699 ts_tablet_manager.cc:1905] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:05:57.313943 13700 ts_tablet_manager.cc:1918] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:05:57.314244 13700 log.cc:1196] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/b42a46f2528645e991bfa293c7c9ade6
I20260929 02:05:57.315491 13700 ts_tablet_manager.cc:1939] T b42a46f2528645e991bfa293c7c9ade6 P 348a62b8c67e465f93ba8a5500720d10: Deleting consensus metadata
I20260929 02:05:57.316892 13112 catalog_manager.cc:4841] TS 348a62b8c67e465f93ba8a5500720d10 (127.11.182.193:33545): tablet b42a46f2528645e991bfa293c7c9ade6 (table test-table [id=92ea36529a034db297eb957e16970c0f]) successfully deleted
I20260929 02:05:57.324606 13699 ts_tablet_manager.cc:1918] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:05:57.324764 13699 log.cc:1196] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/b42a46f2528645e991bfa293c7c9ade6
I20260929 02:05:57.325287 13699 ts_tablet_manager.cc:1939] T b42a46f2528645e991bfa293c7c9ade6 P 13ea0af01ff244c7872b39a951e4bcc0: Deleting consensus metadata
I20260929 02:05:57.325554 13701 ts_tablet_manager.cc:1918] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:05:57.325699 13701 log.cc:1196] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/b42a46f2528645e991bfa293c7c9ade6
I20260929 02:05:57.326169 13701 ts_tablet_manager.cc:1939] T b42a46f2528645e991bfa293c7c9ade6 P 6374faa171184b3994a5308d2eca338e: Deleting consensus metadata
I20260929 02:05:57.326579 13112 catalog_manager.cc:4841] TS 13ea0af01ff244c7872b39a951e4bcc0 (127.11.182.195:45197): tablet b42a46f2528645e991bfa293c7c9ade6 (table test-table [id=92ea36529a034db297eb957e16970c0f]) successfully deleted
I20260929 02:05:57.327947 13110 catalog_manager.cc:4841] TS 6374faa171184b3994a5308d2eca338e (127.11.182.194:34807): tablet b42a46f2528645e991bfa293c7c9ade6 (table test-table [id=92ea36529a034db297eb957e16970c0f]) successfully deleted
Sep 29 02:05:57 dist-test-slave-q06p krb5kdc[13081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:05:57.348301 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.312121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.11.182.193:33545)
0929 02:05:57.319293 (+ 7172us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:57.319301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:57.327310 (+ 8009us) server_negotiation.cc:407] Connection header received
0929 02:05:57.327618 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:57.327626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:57.327722 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:57.327820 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:57.330074 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.331287 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.333276 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.333939 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.334071 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.339960 (+ 5889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:57.339982 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:57.340004 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:57.340061 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:57.344076 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.345380 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.345392 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.345396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.345582 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.346072 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.346078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.346081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.346894 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:57.347247 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:57.347677 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 02:05:57.347911 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7027,"thread_start_us":6901,"threads_started":1}
W20260929 02:05:57.349004 13328 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41914
I20260929 02:05:57.367378 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.351908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.11.182.254:44461)
0929 02:05:57.352122 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:57.352129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:57.353019 (+ 890us) server_negotiation.cc:407] Connection header received
0929 02:05:57.353186 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:57.353190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:57.353259 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:57.353401 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:57.354907 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.355641 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.356993 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.357526 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.357684 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.360493 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:57.360498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:57.360501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:57.360539 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:57.363696 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.364446 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.364452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.364456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.364592 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.365019 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.365026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.365028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.365331 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:57.365647 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:57.365953 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 02:05:57.366184 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 02:05:57.367951 13125 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53178
I20260929 02:05:57.371482 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13197
I20260929 02:05:57.386246 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.372495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.11.182.193:33545)
0929 02:05:57.372753 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:57.372760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:57.373039 (+ 279us) server_negotiation.cc:407] Connection header received
0929 02:05:57.373290 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:57.373296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:57.373379 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:57.373503 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:57.374794 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.375609 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.376938 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.377427 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.377586 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.380208 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:57.380226 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:57.380231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:57.380310 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:57.383378 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.384218 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.384225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.384228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.384385 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.384786 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.384792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.384795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.385066 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:57.385337 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:57.385657 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 02:05:57.385920 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 02:05:57.386817 13328 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41930
W20260929 02:05:57.387490 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13197: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:05:57.387678 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 13197
I20260929 02:05:57.410041 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13337
Sep 29 02:05:57 dist-test-slave-q06p krb5kdc[13081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:05:57.427435 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.410881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.11.182.194:34807)
0929 02:05:57.411097 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:57.411103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:57.411124 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:05:57.411277 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:57.411288 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:57.411375 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:57.411487 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:57.412717 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.413860 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.415346 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.416080 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.416223 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.420468 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:57.420485 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:57.420504 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:57.420545 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:57.424409 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.425252 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.425262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.425266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.425447 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.425788 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.425792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.425796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.426310 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:57.426655 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:57.426822 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 02:05:57.427078 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 02:05:57.428272 13467 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50554
W20260929 02:05:57.429049 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13337: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:05:57.429231 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 13337
I20260929 02:05:57.448148 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13476
Sep 29 02:05:57 dist-test-slave-q06p krb5kdc[13081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:05:57.464396 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:57.448846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.11.182.195:45197)
0929 02:05:57.449326 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:57.449330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:57.449354 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:05:57.449407 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:57.449411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:57.449465 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:57.449560 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:57.451009 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.451697 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.453112 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:57.453590 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:57.453729 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:57.458365 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:57.458380 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:57.458397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:57.458441 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:57.461432 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.462217 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.462224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.462227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.462365 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:57.462725 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:57.462732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:57.462735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:57.463203 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:57.463506 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:57.463791 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 02:05:57.464026 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
W20260929 02:05:57.465052 13607 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39766
W20260929 02:05:57.465812 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13476: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:05:57.466015 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 13476
I20260929 02:05:57.483052 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13094
I20260929 02:05:57.484191 13166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:05:57.598825 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 13094
2026-09-29T02:05:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5057 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:05:57 dist-test-slave-q06p krb5kdc[13712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:05:57 dist-test-slave-q06p krb5kdc[13712](info): set up 2 sockets
Sep 29 02:05:57 dist-test-slave-q06p krb5kdc[13712](info): commencing operation
krb5kdc: starting...
W20260929 02:05:58.748483 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:05:58 dist-test-slave-q06p krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:05:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:05:59.003793 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45287
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:05:59.363663 13725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:05:59.364068 13725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:05:59.364228 13725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:05:59.371848 13725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:05:59.372143 13725 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:05:59.372316 13725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:05:59.372514 13725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:05:59.372606 13725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:05:59.372699 13725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:05:59.383651 13725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:45287
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:05:59.385119 13725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:05:59.386536 13725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:05:59.399178 13733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:59.399313 13730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:05:59.399260 13731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:05:59.399976 13725 server_base.cc:1034] running on GCE node
I20260929 02:05:59.400872 13725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:05:59.402443 13725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:05:59.403734 13725 hybrid_clock.cc:648] HybridClock initialized: now 1790647559403665 us; error 52 us; skew 500 ppm
I20260929 02:05:59.404237 13725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:05:59 dist-test-slave-q06p krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647559, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:05:59.407716 13725 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:05:59.409387 13725 webserver.cc:460] Webserver started at http://127.11.182.254:36767/ using document root <none> and password file <none>
I20260929 02:05:59.410091 13725 fs_manager.cc:359] Metadata directory not provided
I20260929 02:05:59.410179 13725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:05:59.410437 13725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:05:59.413997 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "e62c97e444a946f882182c0d188decff"
format_stamp: "Formatted at 2026-09-29 02:05:59 on dist-test-slave-q06p"
server_key: "6cf91982ed2d91ef161b941ed893d6fd"
server_key_iv: "67a9e7f7fdc29f9b1f26fe31cc87aa38"
server_key_version: "encryptionkey@0"
I20260929 02:05:59.414822 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "e62c97e444a946f882182c0d188decff"
format_stamp: "Formatted at 2026-09-29 02:05:59 on dist-test-slave-q06p"
server_key: "6cf91982ed2d91ef161b941ed893d6fd"
server_key_iv: "67a9e7f7fdc29f9b1f26fe31cc87aa38"
server_key_version: "encryptionkey@0"
I20260929 02:05:59.420507 13725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:05:59.424360 13740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:05:59.426064 13725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:05:59.426285 13725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "e62c97e444a946f882182c0d188decff"
format_stamp: "Formatted at 2026-09-29 02:05:59 on dist-test-slave-q06p"
server_key: "6cf91982ed2d91ef161b941ed893d6fd"
server_key_iv: "67a9e7f7fdc29f9b1f26fe31cc87aa38"
server_key_version: "encryptionkey@0"
I20260929 02:05:59.426546 13725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:05:59.450737 13725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:05:59.455610 13725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:05:59.455938 13725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:05:59.479877 13725 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:45287
I20260929 02:05:59.479890 13801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:45287 every 8 connection(s)
I20260929 02:05:59.481297 13725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:05:59.485620 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 13725
I20260929 02:05:59.485893 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:05:59.486171 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:05:59.486722 11995 external_mini_cluster.cc:1428] Setting key 46d333a8c707bbc53c31be34f2b9fcd7
I20260929 02:05:59.501968 13802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff: Bootstrap starting.
Sep 29 02:05:59 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:05:59.505550 13802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff: Neither blocks nor log segments found. Creating new log.
I20260929 02:05:59.506728 13802 log.cc:824] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff: Log is configured to *not* fsync() on all Append() calls
I20260929 02:05:59.509686 13802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff: No bootstrap required, opened a new log
I20260929 02:05:59.511262 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:05:59.489010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.11.182.254:45287)
0929 02:05:59.489838 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0929 02:05:59.489851 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:05:59.489897 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:05:59.491769 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:05:59.491795 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:05:59.492250 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:05:59.492672 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:05:59.494546 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:59.495939 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:59.497760 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:05:59.498540 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:05:59.498687 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:05:59.503554 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:05:59.503566 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:05:59.503582 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:05:59.503630 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:05:59.506921 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:59.507688 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:59.507701 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:59.507709 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:59.507887 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:05:59.508294 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:05:59.508301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:05:59.508304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:05:59.508828 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:05:59.509128 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 02:05:59.509584 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 02:05:59.510167 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":385,"threads_started":1}
I20260929 02:05:59.517645 13802 raft_consensus.cc:348] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62c97e444a946f882182c0d188decff" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45287 } }
I20260929 02:05:59.518117 13802 raft_consensus.cc:374] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:05:59.518255 13802 raft_consensus.cc:729] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62c97e444a946f882182c0d188decff, State: Initialized, Role: FOLLOWER
I20260929 02:05:59.519016 13802 consensus_queue.cc:260] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62c97e444a946f882182c0d188decff" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45287 } }
I20260929 02:05:59.519254 13802 raft_consensus.cc:388] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:05:59.519361 13802 raft_consensus.cc:482] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:05:59.519507 13802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:05:59.521713 13802 raft_consensus.cc:504] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62c97e444a946f882182c0d188decff" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45287 } }
I20260929 02:05:59.522348 13802 leader_election.cc:302] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e62c97e444a946f882182c0d188decff; no voters:
I20260929 02:05:59.523097 13802 leader_election.cc:288] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:05:59.523237 13807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:05:59.524008 13807 raft_consensus.cc:686] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [term 1 LEADER]: Becoming Leader. State: Replica: e62c97e444a946f882182c0d188decff, State: Running, Role: LEADER
I20260929 02:05:59.524711 13807 consensus_queue.cc:237] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62c97e444a946f882182c0d188decff" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45287 } }
I20260929 02:05:59.525725 13802 sys_catalog.cc:564] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:05:59.528095 13809 sys_catalog.cc:455] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e62c97e444a946f882182c0d188decff. Latest consensus state: current_term: 1 leader_uuid: "e62c97e444a946f882182c0d188decff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62c97e444a946f882182c0d188decff" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45287 } } }
I20260929 02:05:59.528457 13809 sys_catalog.cc:458] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [sys.catalog]: This master's current role is: LEADER
I20260929 02:05:59.529347 13813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:05:59.530542 13808 sys_catalog.cc:455] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e62c97e444a946f882182c0d188decff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62c97e444a946f882182c0d188decff" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45287 } } }
I20260929 02:05:59.531147 13808 sys_catalog.cc:458] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff [sys.catalog]: This master's current role is: LEADER
I20260929 02:05:59.537566 13813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:05:59.548669 13813 catalog_manager.cc:1347] Generated new cluster ID: 70a4a01da24541b0940496c9daa37146
I20260929 02:05:59.548810 13813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:05:59.577067 13813 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:05:59.578550 13813 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:05:59.588150 13813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff: Generated new TSK 0
I20260929 02:05:59.589104 13813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:05:59.652443 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45287
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:00.045271 13828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:00.045672 13828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:00.045857 13828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:00.054018 13828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:00.054975 13828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:00.055184 13828 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:06:00.055316 13828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:06:00.055464 13828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:00.066552 13828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.182.254:45287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:00.068008 13828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:00.069542 13828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:00.088061 13836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:00.088081 13834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:00.088303 13833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:00.090404 13828 server_base.cc:1034] running on GCE node
I20260929 02:06:00.091372 13828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:00.092665 13828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:00.094008 13828 hybrid_clock.cc:648] HybridClock initialized: now 1790647560093945 us; error 60 us; skew 500 ppm
I20260929 02:06:00.094470 13828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:00 dist-test-slave-q06p krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647560, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:00.099159 13828 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:06:00.100936 13828 webserver.cc:460] Webserver started at http://127.11.182.193:39539/ using document root <none> and password file <none>
I20260929 02:06:00.101723 13828 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:00.101889 13828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:00.102236 13828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:00.105732 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "90862fe507bb4dbd800ff84e869c3914"
format_stamp: "Formatted at 2026-09-29 02:06:00 on dist-test-slave-q06p"
server_key: "5dfe8ab03dcfead9a28bb8f2c5f10b9a"
server_key_iv: "0d0465dcf3f580f6663c7277d15bd323"
server_key_version: "encryptionkey@0"
I20260929 02:06:00.106675 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "90862fe507bb4dbd800ff84e869c3914"
format_stamp: "Formatted at 2026-09-29 02:06:00 on dist-test-slave-q06p"
server_key: "5dfe8ab03dcfead9a28bb8f2c5f10b9a"
server_key_iv: "0d0465dcf3f580f6663c7277d15bd323"
server_key_version: "encryptionkey@0"
I20260929 02:06:00.114636 13828 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260929 02:06:00.119035 13843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:00.120784 13828 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:00.121031 13828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "90862fe507bb4dbd800ff84e869c3914"
format_stamp: "Formatted at 2026-09-29 02:06:00 on dist-test-slave-q06p"
server_key: "5dfe8ab03dcfead9a28bb8f2c5f10b9a"
server_key_iv: "0d0465dcf3f580f6663c7277d15bd323"
server_key_version: "encryptionkey@0"
I20260929 02:06:00.121461 13828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:00.137089 13828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:00.142716 13828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:00.143102 13828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:00.147311 13828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:00.147677 13828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 02:06:00.148074 13828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:00.148255 13828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 02:06:00 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647560, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:00.177208 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:00.147701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:47959 (local address 127.11.182.254:45287)
0929 02:06:00.148291 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:00.148296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:00.150055 (+ 1759us) server_negotiation.cc:407] Connection header received
0929 02:06:00.153086 (+ 3031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:00.153092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:00.153205 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:00.153360 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:00.155968 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.157369 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.160601 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.161100 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.161271 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:00.167777 (+ 6506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:00.167785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:00.167789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:00.167840 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:00.172456 (+ 4616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:00.173531 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:00.173541 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:00.173547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:00.173755 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:00.174367 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:00.174374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:00.174381 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:00.174892 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:00.175087 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:00.176526 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0929 02:06:00.176802 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":343,"threads_started":1}
I20260929 02:06:00.178769 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:00.148895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45287 (local address 127.11.182.193:47959)
0929 02:06:00.149836 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:00.149890 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:00.152359 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:00.153666 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:00.153678 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:00.154603 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:00.155698 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:00.155740 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.157625 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.157633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:00.160377 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:00.160397 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.161513 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.161525 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:00.161934 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:00.163051 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:00.163077 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:00.167290 (+ 4213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:00.172747 (+ 5457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:00.172757 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:00.172775 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:00.173344 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:00.173943 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:00.173947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:00.173949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:00.174226 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:00.175081 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:00.175094 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:00.175874 (+ 780us) client_negotiation.cc:769] Sending connection context
0929 02:06:00.176371 (+ 497us) client_negotiation.cc:240] Negotiation successful
0929 02:06:00.177056 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":414,"threads_started":1}
I20260929 02:06:00.206270 13828 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:38251
I20260929 02:06:00.206313 13963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:38251 every 8 connection(s)
I20260929 02:06:00.207852 13828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:00.208177 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:00.190119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:57561 (local address 127.11.182.254:45287)
0929 02:06:00.190380 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:00.190391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:00.190993 (+ 602us) server_negotiation.cc:407] Connection header received
0929 02:06:00.191262 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:00.191271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:00.191365 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:00.191478 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:00.193293 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.194208 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.197185 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.197735 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.198072 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:00.201411 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:00.201420 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:00.201427 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:00.201483 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:00.204816 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:00.205852 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:00.205867 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:00.205875 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:00.206012 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:00.206483 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:00.206488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:00.206491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:00.206826 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:00.207055 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:00.207528 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 02:06:00.207779 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:06:00.208389 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:00.190059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45287 (local address 127.11.182.193:57561)
0929 02:06:00.190822 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:00.190874 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:00.191101 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:00.191539 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:00.191547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:00.192358 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:00.192879 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:00.192898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.194386 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.194394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:00.196711 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:00.196735 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.198204 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.198217 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:00.198526 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:00.199484 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:00.199514 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:00.201108 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:00.205036 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:00.205042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:00.205047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:00.205697 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:00.206149 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:00.206159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:00.206163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:00.206311 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:00.207007 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:00.207012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:00.207225 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 02:06:00.207416 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 02:06:00.207795 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":374,"threads_started":1}
I20260929 02:06:00.208624 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 13828
I20260929 02:06:00.208886 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:00.209939 11995 external_mini_cluster.cc:1428] Setting key 77d4a09a17e5c0f388a192d8efdb21b0
I20260929 02:06:00.214004 13964 heartbeater.cc:347] Connected to a master server at 127.11.182.254:45287
I20260929 02:06:00.214483 13964 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:00.215842 13964 heartbeater.cc:511] Master 127.11.182.254:45287 requested a full tablet report, sending...
I20260929 02:06:00.219525 13756 ts_manager.cc:194] Registered new tserver with Master: 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251)
I20260929 02:06:00.221246 13756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:47959
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:06:00.263962 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45287
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:00.626892 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:00.627354 13968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:00.627548 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:00.636227 13968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:00.636416 13968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:00.636610 13968 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:06:00.636679 13968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:06:00.636772 13968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:00.647804 13968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.182.254:45287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:00.649312 13968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:00.650766 13968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:00.666549 13974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:00.666575 13973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:00.666610 13976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:00.667248 13968 server_base.cc:1034] running on GCE node
I20260929 02:06:00.668013 13968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:00.669145 13968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:00.670473 13968 hybrid_clock.cc:648] HybridClock initialized: now 1790647560670397 us; error 70 us; skew 500 ppm
I20260929 02:06:00.671041 13968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:00 dist-test-slave-q06p krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647560, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:00.674571 13968 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:06:00.676456 13968 webserver.cc:460] Webserver started at http://127.11.182.194:45333/ using document root <none> and password file <none>
I20260929 02:06:00.677234 13968 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:00.677352 13968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:00.677723 13968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:00.681329 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "ee20f6423bac4a23876e09f27c1f5b8b"
format_stamp: "Formatted at 2026-09-29 02:06:00 on dist-test-slave-q06p"
server_key: "1f3c40cda2e8463a5726a03d7554b42f"
server_key_iv: "a38e03ec7fa138c0650823b8506c2218"
server_key_version: "encryptionkey@0"
I20260929 02:06:00.682255 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "ee20f6423bac4a23876e09f27c1f5b8b"
format_stamp: "Formatted at 2026-09-29 02:06:00 on dist-test-slave-q06p"
server_key: "1f3c40cda2e8463a5726a03d7554b42f"
server_key_iv: "a38e03ec7fa138c0650823b8506c2218"
server_key_version: "encryptionkey@0"
I20260929 02:06:00.688637 13968 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 02:06:00.692839 13983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:00.694602 13968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:06:00.695116 13968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "ee20f6423bac4a23876e09f27c1f5b8b"
format_stamp: "Formatted at 2026-09-29 02:06:00 on dist-test-slave-q06p"
server_key: "1f3c40cda2e8463a5726a03d7554b42f"
server_key_iv: "a38e03ec7fa138c0650823b8506c2218"
server_key_version: "encryptionkey@0"
I20260929 02:06:00.695351 13968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:00.709285 13968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:00.714221 13968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:00.714583 13968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:00.717639 13968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:00.717746 13968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:00.717895 13968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:00.718014 13968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:06:00 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647560, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:00.745755 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:00.718678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:60107 (local address 127.11.182.254:45287)
0929 02:06:00.719191 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:00.719198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:00.720525 (+ 1327us) server_negotiation.cc:407] Connection header received
0929 02:06:00.722596 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:00.722602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:00.722699 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:00.722847 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:00.726472 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.727637 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.731248 (+ 3611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.731930 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.732091 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:00.738174 (+ 6083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:00.738179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:00.738183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:00.738224 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:00.742009 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:00.742958 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:00.742969 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:00.742971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:00.743109 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:00.743658 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:00.743663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:00.743672 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:00.744083 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:00.744306 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:00.745072 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 02:06:00.745346 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":264,"threads_started":1}
I20260929 02:06:00.747071 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:00.719391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45287 (local address 127.11.182.194:60107)
0929 02:06:00.720325 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:00.720374 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:00.722274 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:00.723389 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:00.723403 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:00.724535 (+ 1132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:00.726164 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:00.726213 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.727858 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.727868 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:00.730980 (+ 3112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:00.731004 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.732273 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.732284 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:00.732633 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:00.733735 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:00.733913 (+ 178us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:00.737865 (+ 3952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:00.742203 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:00.742215 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:00.742236 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:00.742768 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:00.743249 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:00.743255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:00.743258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:00.743514 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:00.744232 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:00.744247 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:00.744674 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 02:06:00.744964 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 02:06:00.745557 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":387,"threads_started":1}
I20260929 02:06:00.775437 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:00.755794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:41315 (local address 127.11.182.254:45287)
0929 02:06:00.755972 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:00.755976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:00.757228 (+ 1252us) server_negotiation.cc:407] Connection header received
0929 02:06:00.757456 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:00.757464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:00.757569 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:00.757679 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:00.760201 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.760974 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.762437 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.763005 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.763155 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:00.766073 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:00.766084 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:00.766091 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:00.766145 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:00.770675 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:00.772254 (+ 1579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:00.772265 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:00.772272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:00.772488 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:00.773089 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:00.773098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:00.773101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:00.773671 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:00.773967 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:00.774538 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 02:06:00.774985 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:06:00.775614 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:00.755733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45287 (local address 127.11.182.194:41315)
0929 02:06:00.757075 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:00.757095 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:00.757307 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:00.758082 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:00.758089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:00.759060 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:00.759951 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:00.759978 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.761170 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.761175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:00.762240 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:00.762253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:00.763234 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:00.763240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:00.763438 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:00.764196 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:00.764216 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:00.765691 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:00.771002 (+ 5311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:00.771012 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:00.771018 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:00.772039 (+ 1021us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:00.772676 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:00.772685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:00.772695 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:00.772931 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:00.773936 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:00.773945 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:00.774193 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 02:06:00.774335 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 02:06:00.774626 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"thread_start_us":1014,"threads_started":1}
I20260929 02:06:00.783922 13968 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:40007
I20260929 02:06:00.784022 14103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:40007 every 8 connection(s)
I20260929 02:06:00.785316 13968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:00.787416 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 13968
I20260929 02:06:00.787606 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:00.788091 11995 external_mini_cluster.cc:1428] Setting key 35166ae788c26c107d0c8a175f7e9e05
I20260929 02:06:00.788061 14104 heartbeater.cc:347] Connected to a master server at 127.11.182.254:45287
I20260929 02:06:00.788335 14104 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:00.789050 14104 heartbeater.cc:511] Master 127.11.182.254:45287 requested a full tablet report, sending...
I20260929 02:06:00.790632 13756 ts_manager.cc:194] Registered new tserver with Master: ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007)
I20260929 02:06:00.791993 13756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:60107
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:06:00.835767 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45287
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:01.205233 14108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:01.205667 14108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:01.205896 14108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:01.216274 14108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:01.216420 14108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:01.216509 14108 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:06:01.216563 14108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:06:01.216666 14108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:01.224651 13964 heartbeater.cc:503] Master 127.11.182.254:45287 was elected leader, sending a full tablet report...
I20260929 02:06:01.228163 14108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.182.254:45287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:01.229554 14108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:01.231174 14108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:01.246869 14116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:01.246884 14114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:01.246747 14113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:01.247759 14108 server_base.cc:1034] running on GCE node
I20260929 02:06:01.248624 14108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:01.249830 14108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:01.251117 14108 hybrid_clock.cc:648] HybridClock initialized: now 1790647561251053 us; error 83 us; skew 500 ppm
I20260929 02:06:01.251488 14108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:01 dist-test-slave-q06p krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647561, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:01.255028 14108 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:06:01.256927 14108 webserver.cc:460] Webserver started at http://127.11.182.195:38501/ using document root <none> and password file <none>
I20260929 02:06:01.257791 14108 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:01.257923 14108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:01.258204 14108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:01.262120 14108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "48df04fc8dc641e49c18563a9ff7fa15"
format_stamp: "Formatted at 2026-09-29 02:06:01 on dist-test-slave-q06p"
server_key: "98a8d9bfd4bbf0182fc7cd6eb7126920"
server_key_iv: "7e68ddd3ba18deb620247029f1e63aea"
server_key_version: "encryptionkey@0"
I20260929 02:06:01.263082 14108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "48df04fc8dc641e49c18563a9ff7fa15"
format_stamp: "Formatted at 2026-09-29 02:06:01 on dist-test-slave-q06p"
server_key: "98a8d9bfd4bbf0182fc7cd6eb7126920"
server_key_iv: "7e68ddd3ba18deb620247029f1e63aea"
server_key_version: "encryptionkey@0"
I20260929 02:06:01.269748 14108 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 02:06:01.274561 14123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:01.276364 14108 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:01.276571 14108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "48df04fc8dc641e49c18563a9ff7fa15"
format_stamp: "Formatted at 2026-09-29 02:06:01 on dist-test-slave-q06p"
server_key: "98a8d9bfd4bbf0182fc7cd6eb7126920"
server_key_iv: "7e68ddd3ba18deb620247029f1e63aea"
server_key_version: "encryptionkey@0"
I20260929 02:06:01.276798 14108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:01.291107 14108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:01.296465 14108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:01.296845 14108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:01.300365 14108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:01.300616 14108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 02:06:01.300788 14108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:01.300971 14108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:06:01 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647561, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:01.329286 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.300713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:37469 (local address 127.11.182.254:45287)
0929 02:06:01.301308 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.301320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.303193 (+ 1873us) server_negotiation.cc:407] Connection header received
0929 02:06:01.306090 (+ 2897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.306098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.306210 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.306372 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:01.308730 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.309770 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.312485 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.313146 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.313313 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.320169 (+ 6856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:01.320176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:01.320179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:01.320216 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:01.323937 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:01.325297 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:01.325304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:01.325307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:01.325470 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:01.326501 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:01.326509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:01.326515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:01.327004 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:01.327203 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.328675 (+ 1472us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.328966 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":230,"threads_started":1}
I20260929 02:06:01.331300 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.301533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45287 (local address 127.11.182.195:37469)
0929 02:06:01.302971 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.303032 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.305495 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.306772 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.306789 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.307537 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:01.308447 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.308469 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.310014 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.310023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.312241 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.312266 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.313432 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.313439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.313739 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.314878 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:01.314929 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:01.319761 (+ 4832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:01.324179 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:01.324187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:01.324203 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:01.325087 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:01.325666 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:01.325675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:01.325681 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:01.326148 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:01.327178 (+ 1030us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:01.327188 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:01.328029 (+ 841us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.328512 (+ 483us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.329213 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":822,"threads_started":1}
I20260929 02:06:01.355553 14108 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:40173
I20260929 02:06:01.355587 14243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:40173 every 8 connection(s)
I20260929 02:06:01.357340 14108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:01.358255 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.341318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45287 (local address 127.11.182.195:38037)
0929 02:06:01.342003 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.342032 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.342237 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.342714 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.342719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.343474 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:01.344054 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.344066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.345163 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.345171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.346763 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.346778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.347811 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.347821 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.348117 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.349017 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:01.349042 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:01.350746 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:01.355110 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:01.355116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:01.355121 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:01.355702 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:01.356198 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:01.356202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:01.356204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:01.356338 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:01.357082 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:01.357086 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:01.357254 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.357385 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.357738 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":336,"threads_started":1}
I20260929 02:06:01.358239 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.341241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:38037 (local address 127.11.182.254:45287)
0929 02:06:01.341501 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.341511 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.342148 (+ 637us) server_negotiation.cc:407] Connection header received
0929 02:06:01.342477 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.342481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.342570 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.342684 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:01.344235 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.345021 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.346976 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.347569 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.347720 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.351118 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:01.351127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:01.351132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:01.351205 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:01.354910 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:01.355885 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:01.355892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:01.355895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:01.356069 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:01.356495 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:01.356500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:01.356503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:01.356939 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:01.357112 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.357541 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.357782 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:06:01.359748 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 14108
I20260929 02:06:01.359941 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:01.360472 11995 external_mini_cluster.cc:1428] Setting key b282f395fe91da3205ede7449d38430a
I20260929 02:06:01.361071 14244 heartbeater.cc:347] Connected to a master server at 127.11.182.254:45287
I20260929 02:06:01.361325 14244 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:01.362193 14244 heartbeater.cc:511] Master 127.11.182.254:45287 requested a full tablet report, sending...
I20260929 02:06:01.363803 13756 ts_manager.cc:194] Registered new tserver with Master: 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195:40173)
I20260929 02:06:01.364797 13756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:37469
I20260929 02:06:01.365368 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:06:01 dist-test-slave-q06p krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:06:01 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:01.416600 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.400923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.11.182.254:45287)
0929 02:06:01.401262 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.401270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.401640 (+ 370us) server_negotiation.cc:407] Connection header received
0929 02:06:01.401902 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.401906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.401973 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.402090 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:01.403692 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.404437 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.405707 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.406113 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.406254 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.410971 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:01.410977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:01.410980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:01.411024 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:01.413870 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:01.414602 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:01.414608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:01.414613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:01.414751 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:01.415183 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:01.415190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:01.415193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:01.415486 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:01.415742 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.416022 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.416248 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260929 02:06:01.421687 13756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:01.429112 13756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:01.453974 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.443067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.11.182.195:40173)
0929 02:06:01.443786 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.443797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.444398 (+ 601us) server_negotiation.cc:407] Connection header received
0929 02:06:01.444728 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.444739 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.445063 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.445313 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:01.446426 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.447777 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.450927 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.452526 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.452746 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.452958 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.453306 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.453501 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":334,"threads_started":1}
I20260929 02:06:01.454031 14257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.442893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:40173 (local address 127.0.0.1:36226)
0929 02:06:01.444195 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.444254 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.444457 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.445655 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.445671 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.445695 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:01.446164 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.446181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.447914 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.447923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.450643 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.450671 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.452751 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.452757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.453036 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.453090 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.453258 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.453501 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":1,"thread_start_us":874,"threads_started":1}
I20260929 02:06:01.457638 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.446051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38251 (local address 127.0.0.1:49526)
0929 02:06:01.448426 (+ 2375us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.448445 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.448562 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.449965 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.449970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.449994 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:01.450455 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.450474 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.452459 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.452469 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.454960 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.454981 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.456794 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.456800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.456978 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.457013 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.457106 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.457246 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2235,"mutex_wait_us":28,"thread_start_us":2048,"threads_started":1}
I20260929 02:06:01.458364 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.445684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.11.182.193:38251)
0929 02:06:01.449086 (+ 3402us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.449095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.449126 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:06:01.449233 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.449243 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.449530 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.449753 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:01.450694 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.452318 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.455154 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.456591 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.456792 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.456989 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.457903 (+ 914us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.458076 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3204,"thread_start_us":3081,"threads_started":1}
I20260929 02:06:01.458704 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.444693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:40007 (local address 127.0.0.1:42730)
0929 02:06:01.445284 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.445302 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.445480 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.450164 (+ 4684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.450170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.450201 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:01.450732 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.450744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.452239 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.452247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.456158 (+ 3911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.456171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.457907 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.457915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.458083 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.458114 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.458178 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.458314 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":1481,"thread_start_us":321,"threads_started":1}
I20260929 02:06:01.459671 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.445017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.11.182.194:40007)
0929 02:06:01.449532 (+ 4515us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.449541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.449574 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:06:01.449677 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.449683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.449943 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.450965 (+ 1022us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:01.451185 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.452086 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.456331 (+ 4245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.457659 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.459000 (+ 1341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.459173 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.459242 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.459415 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4306,"thread_start_us":4180,"threads_started":1}
I20260929 02:06:01.458690 14174 tablet_service.cc:1467] Processing CreateTablet for tablet a6c6a33774564cd18888770bd2ed7146 (DEFAULT_TABLE table=test-table [id=dce844efc16a4987927172661830f367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:01.462713 13894 tablet_service.cc:1467] Processing CreateTablet for tablet a6c6a33774564cd18888770bd2ed7146 (DEFAULT_TABLE table=test-table [id=dce844efc16a4987927172661830f367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:01.464252 14174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c6a33774564cd18888770bd2ed7146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:01.464485 13894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c6a33774564cd18888770bd2ed7146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:01.463982 14034 tablet_service.cc:1467] Processing CreateTablet for tablet a6c6a33774564cd18888770bd2ed7146 (DEFAULT_TABLE table=test-table [id=dce844efc16a4987927172661830f367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:01.465602 14034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c6a33774564cd18888770bd2ed7146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:01.476652 14263 tablet_bootstrap.cc:492] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: Bootstrap starting.
I20260929 02:06:01.480381 14263 tablet_bootstrap.cc:654] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:01.481802 14263 log.cc:824] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:01.487140 14264 tablet_bootstrap.cc:492] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: Bootstrap starting.
I20260929 02:06:01.487140 14265 tablet_bootstrap.cc:492] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: Bootstrap starting.
I20260929 02:06:01.491861 14265 tablet_bootstrap.cc:654] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:01.494092 14264 tablet_bootstrap.cc:654] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:01.494771 14265 log.cc:824] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:01.495908 14264 log.cc:824] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:01.502883 14263 tablet_bootstrap.cc:492] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: No bootstrap required, opened a new log
I20260929 02:06:01.503113 14265 tablet_bootstrap.cc:492] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: No bootstrap required, opened a new log
I20260929 02:06:01.503219 14263 ts_tablet_manager.cc:1397] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.012s
I20260929 02:06:01.503547 14265 ts_tablet_manager.cc:1397] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20260929 02:06:01.506332 14264 tablet_bootstrap.cc:492] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: No bootstrap required, opened a new log
I20260929 02:06:01.506834 14264 ts_tablet_manager.cc:1397] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20260929 02:06:01.513058 14263 raft_consensus.cc:348] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } }
I20260929 02:06:01.513509 14263 raft_consensus.cc:374] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:01.513660 14263 raft_consensus.cc:729] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48df04fc8dc641e49c18563a9ff7fa15, State: Initialized, Role: FOLLOWER
I20260929 02:06:01.514343 14263 consensus_queue.cc:260] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } }
I20260929 02:06:01.514582 14265 raft_consensus.cc:348] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } }
I20260929 02:06:01.515177 14265 raft_consensus.cc:374] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:01.515594 14244 heartbeater.cc:503] Master 127.11.182.254:45287 was elected leader, sending a full tablet report...
I20260929 02:06:01.516222 14263 ts_tablet_manager.cc:1428] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260929 02:06:01.515321 14265 raft_consensus.cc:729] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee20f6423bac4a23876e09f27c1f5b8b, State: Initialized, Role: FOLLOWER
I20260929 02:06:01.517191 14265 consensus_queue.cc:260] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } }
I20260929 02:06:01.518409 14104 heartbeater.cc:503] Master 127.11.182.254:45287 was elected leader, sending a full tablet report...
I20260929 02:06:01.518271 14264 raft_consensus.cc:348] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } }
I20260929 02:06:01.518718 14264 raft_consensus.cc:374] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:01.518819 14264 raft_consensus.cc:729] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90862fe507bb4dbd800ff84e869c3914, State: Initialized, Role: FOLLOWER
I20260929 02:06:01.519143 14265 ts_tablet_manager.cc:1428] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260929 02:06:01.519569 14264 consensus_queue.cc:260] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } }
I20260929 02:06:01.521426 14264 ts_tablet_manager.cc:1428] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260929 02:06:01.537652 14105 tablet.cc:2369] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:06:01.557106 14269 raft_consensus.cc:482] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:01.557411 14269 raft_consensus.cc:504] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } }
I20260929 02:06:01.559115 14269 leader_election.cc:288] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007), 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251)
I20260929 02:06:01.565835 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.559658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:47369 (local address 127.11.182.194:40007)
0929 02:06:01.559843 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.559851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.559873 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:06:01.559948 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.559953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.560043 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.560145 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:01.560652 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.561669 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.563743 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.564951 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.565141 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.565199 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.565536 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.565639 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:06:01.565979 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.559405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:40007 (local address 127.11.182.195:47369)
0929 02:06:01.559647 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.559669 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.559804 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.560265 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.560273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.560309 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:01.560505 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.560513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.561832 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.561838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.563567 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.563581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.565158 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.565164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.565367 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.565405 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.565518 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.565670 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:06:01.567239 14059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c6a33774564cd18888770bd2ed7146" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" is_pre_election: true
I20260929 02:06:01.567613 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.559770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38251 (local address 127.11.182.195:44521)
0929 02:06:01.560368 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.560381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.560527 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.560879 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.560884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.560922 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:01.561270 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.561290 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.562825 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.562829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.564953 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.564968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.566639 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.566645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.566885 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.566917 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.567016 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.567183 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":321,"threads_started":1}
I20260929 02:06:01.567638 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.559921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:44521 (local address 127.11.182.193:38251)
0929 02:06:01.560161 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.560167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.560474 (+ 307us) server_negotiation.cc:407] Connection header received
0929 02:06:01.560707 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.560710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.560769 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.560839 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:01.561534 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.562636 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.565154 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.566416 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.566619 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.566690 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.567075 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.567281 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:06:01.567844 14059 raft_consensus.cc:2454] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 0.
I20260929 02:06:01.568560 14127 leader_election.cc:302] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48df04fc8dc641e49c18563a9ff7fa15, ee20f6423bac4a23876e09f27c1f5b8b; no voters:
I20260929 02:06:01.569017 14269 raft_consensus.cc:2781] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:01.568907 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c6a33774564cd18888770bd2ed7146" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90862fe507bb4dbd800ff84e869c3914" is_pre_election: true
I20260929 02:06:01.569212 14269 raft_consensus.cc:482] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:01.569351 14269 raft_consensus.cc:3037] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:01.569481 13914 raft_consensus.cc:2454] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 0.
I20260929 02:06:01.571743 14269 raft_consensus.cc:504] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } }
I20260929 02:06:01.572353 14269 leader_election.cc:288] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 election: Requested vote from peers ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007), 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251)
I20260929 02:06:01.572975 14059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c6a33774564cd18888770bd2ed7146" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b"
I20260929 02:06:01.573099 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c6a33774564cd18888770bd2ed7146" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90862fe507bb4dbd800ff84e869c3914"
I20260929 02:06:01.573208 14059 raft_consensus.cc:3037] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:01.573374 13914 raft_consensus.cc:3037] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:01.575644 14059 raft_consensus.cc:2454] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 1.
I20260929 02:06:01.575752 13914 raft_consensus.cc:2454] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 1.
I20260929 02:06:01.576196 14127 leader_election.cc:302] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48df04fc8dc641e49c18563a9ff7fa15, ee20f6423bac4a23876e09f27c1f5b8b; no voters:
I20260929 02:06:01.576543 14269 raft_consensus.cc:2781] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:01.577057 14269 raft_consensus.cc:686] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 LEADER]: Becoming Leader. State: Replica: 48df04fc8dc641e49c18563a9ff7fa15, State: Running, Role: LEADER
I20260929 02:06:01.577540 14269 consensus_queue.cc:237] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } }
I20260929 02:06:01.583010 13756 catalog_manager.cc:5495] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195). New cstate: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } health_report { overall_health: UNKNOWN } } }
W20260929 02:06:01.610994 14245 tablet.cc:2369] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:06:01 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647559, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:01.645288 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.629817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.11.182.254:45287)
0929 02:06:01.630042 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.630052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.630818 (+ 766us) server_negotiation.cc:407] Connection header received
0929 02:06:01.631196 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.631201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.631270 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.631373 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:01.632576 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.633405 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.634898 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.635439 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.635543 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.639262 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:01.639267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:01.639270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:01.639304 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:01.642407 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:01.643319 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:01.643324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:01.643326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:01.643464 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:01.643828 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:01.643833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:01.643835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:01.644153 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:01.644269 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.644668 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.644951 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":70}
I20260929 02:06:01.645320 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.629751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45287 (local address 127.0.0.1:37546)
0929 02:06:01.630703 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.630720 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.630930 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.631445 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.631448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.632149 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:01.632424 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.632433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.633537 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.633541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.634635 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.634647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.635646 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.635651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.635772 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.636584 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:01.636602 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:01.638919 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:01.642578 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:01.642592 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:01.642598 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:01.643193 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:01.643563 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:01.643566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:01.643569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:01.643694 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:01.644274 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:01.644281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:01.644432 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.644560 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.644821 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":637,"threads_started":1}
I20260929 02:06:01.651108 13756 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:37546:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:06:01.652181 13756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:01.660336 14174 tablet_service.cc:1467] Processing CreateTablet for tablet 224a156c9b8d4d158176b63dc30e9043 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3542dac5e8bc46b5975dabeab7e92047]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:06:01.660248 13894 tablet_service.cc:1467] Processing CreateTablet for tablet 224a156c9b8d4d158176b63dc30e9043 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3542dac5e8bc46b5975dabeab7e92047]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:06:01.660248 14034 tablet_service.cc:1467] Processing CreateTablet for tablet 224a156c9b8d4d158176b63dc30e9043 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3542dac5e8bc46b5975dabeab7e92047]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:06:01.661227 14174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224a156c9b8d4d158176b63dc30e9043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:01.661237 13894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224a156c9b8d4d158176b63dc30e9043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:01.661237 14034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224a156c9b8d4d158176b63dc30e9043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:01.666260 14263 tablet_bootstrap.cc:492] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: Bootstrap starting.
I20260929 02:06:01.667779 14263 tablet_bootstrap.cc:654] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:01.668752 14265 tablet_bootstrap.cc:492] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: Bootstrap starting.
I20260929 02:06:01.669123 14264 tablet_bootstrap.cc:492] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: Bootstrap starting.
I20260929 02:06:01.670253 14263 tablet_bootstrap.cc:492] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: No bootstrap required, opened a new log
I20260929 02:06:01.670497 14263 ts_tablet_manager.cc:1397] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:01.670522 14265 tablet_bootstrap.cc:654] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:01.670704 14264 tablet_bootstrap.cc:654] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:01.671306 14263 raft_consensus.cc:348] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.671530 14263 raft_consensus.cc:374] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:01.671689 14263 raft_consensus.cc:729] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48df04fc8dc641e49c18563a9ff7fa15, State: Initialized, Role: FOLLOWER
I20260929 02:06:01.672016 14263 consensus_queue.cc:260] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.672551 14269 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.672855 14269 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:01.673753 14263 ts_tablet_manager.cc:1428] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:01.674362 14265 tablet_bootstrap.cc:492] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: No bootstrap required, opened a new log
I20260929 02:06:01.674546 14265 ts_tablet_manager.cc:1397] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260929 02:06:01.675489 14265 raft_consensus.cc:348] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.675755 14264 tablet_bootstrap.cc:492] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: No bootstrap required, opened a new log
I20260929 02:06:01.675829 14265 raft_consensus.cc:374] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:01.675985 14264 ts_tablet_manager.cc:1397] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260929 02:06:01.676126 14265 raft_consensus.cc:729] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee20f6423bac4a23876e09f27c1f5b8b, State: Initialized, Role: FOLLOWER
I20260929 02:06:01.676518 14265 consensus_queue.cc:260] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.677196 14265 ts_tablet_manager.cc:1428] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:01.677037 14270 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.677428 14270 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:01.677708 14264 raft_consensus.cc:348] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.677974 14264 raft_consensus.cc:374] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:01.678078 14264 raft_consensus.cc:729] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90862fe507bb4dbd800ff84e869c3914, State: Initialized, Role: FOLLOWER
I20260929 02:06:01.678323 14264 consensus_queue.cc:260] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.679042 14264 ts_tablet_manager.cc:1428] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:01.678912 14271 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.679316 14271 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:01.685973 14269 raft_consensus.cc:482] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:01.686161 14269 raft_consensus.cc:504] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.686743 14269 leader_election.cc:288] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251), ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007)
I20260929 02:06:01.687251 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224a156c9b8d4d158176b63dc30e9043" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90862fe507bb4dbd800ff84e869c3914" is_pre_election: true
I20260929 02:06:01.687328 14059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224a156c9b8d4d158176b63dc30e9043" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" is_pre_election: true
I20260929 02:06:01.687486 13914 raft_consensus.cc:2454] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 0.
I20260929 02:06:01.687646 14059 raft_consensus.cc:2454] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 0.
I20260929 02:06:01.688004 14127 leader_election.cc:302] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48df04fc8dc641e49c18563a9ff7fa15, 90862fe507bb4dbd800ff84e869c3914; no voters:
I20260929 02:06:01.688269 14269 raft_consensus.cc:2781] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:01.688377 14269 raft_consensus.cc:482] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:01.688493 14269 raft_consensus.cc:3037] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:01.690768 14269 raft_consensus.cc:504] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.691392 14269 leader_election.cc:288] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 election: Requested vote from peers 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251), ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007)
I20260929 02:06:01.691908 14059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224a156c9b8d4d158176b63dc30e9043" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b"
I20260929 02:06:01.691910 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224a156c9b8d4d158176b63dc30e9043" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90862fe507bb4dbd800ff84e869c3914"
I20260929 02:06:01.692140 14059 raft_consensus.cc:3037] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:01.692152 13914 raft_consensus.cc:3037] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:01.694329 13914 raft_consensus.cc:2454] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 1.
I20260929 02:06:01.694656 14059 raft_consensus.cc:2454] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 1.
I20260929 02:06:01.694826 14127 leader_election.cc:302] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48df04fc8dc641e49c18563a9ff7fa15, 90862fe507bb4dbd800ff84e869c3914; no voters:
I20260929 02:06:01.695178 14269 raft_consensus.cc:2781] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:01.695374 14269 raft_consensus.cc:686] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 LEADER]: Becoming Leader. State: Replica: 48df04fc8dc641e49c18563a9ff7fa15, State: Running, Role: LEADER
I20260929 02:06:01.695641 14269 consensus_queue.cc:237] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.697619 14273 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: TxnStatusTablet state changed. Reason: New leader 48df04fc8dc641e49c18563a9ff7fa15. Latest consensus state: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.698076 14273 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:06:01.698796 14285 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:06:01.699126 14285 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:06:01.699188 13757 catalog_manager.cc:5495] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195). New cstate: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } health_report { overall_health: HEALTHY } } }
I20260929 02:06:01.719239 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.712437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.11.182.195:40173)
0929 02:06:01.712621 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.712625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.712644 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:06:01.712905 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.712910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.712987 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.713115 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:01.713566 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.714430 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.716501 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.717130 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.717247 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.717932 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:01.718545 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:01.718677 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.718825 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.719025 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:06:01.719332 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.712191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:40173 (local address 127.0.0.1:36236)
0929 02:06:01.712440 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.712461 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.712610 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.713166 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.713171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.713184 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:06:01.713372 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.713385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.714578 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.714584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.716308 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.716328 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.717318 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.717326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.717575 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.717663 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:06:01.718671 (+ 1008us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:06:01.718695 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.718793 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.719001 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 02:06:01.721159 13965 tablet.cc:2369] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:06:01.729419 14059 raft_consensus.cc:1261] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Refusing update from remote peer 48df04fc8dc641e49c18563a9ff7fa15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:01.729480 13914 raft_consensus.cc:1261] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Refusing update from remote peer 48df04fc8dc641e49c18563a9ff7fa15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:01.730643 14269 consensus_queue.cc:1035] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:01.731015 14273 consensus_queue.cc:1035] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:01.744858 14271 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: TxnStatusTablet state changed. Reason: New leader 48df04fc8dc641e49c18563a9ff7fa15. Latest consensus state: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.745215 14271 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:01.757155 14271 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.757771 14271 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P 90862fe507bb4dbd800ff84e869c3914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:01.763659 14269 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.764007 14269 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:06:01.764412 14270 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: TxnStatusTablet state changed. Reason: New leader 48df04fc8dc641e49c18563a9ff7fa15. Latest consensus state: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.764698 14270 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:01.772189 14269 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.773069 14269 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P 48df04fc8dc641e49c18563a9ff7fa15: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:06:01.777576 14270 tablet_replica.cc:439] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } } }
I20260929 02:06:01.777902 14270 tablet_replica.cc:442] T 224a156c9b8d4d158176b63dc30e9043 P ee20f6423bac4a23876e09f27c1f5b8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:01.785904 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.776575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.11.182.195:40173)
0929 02:06:01.776804 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.776815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.777411 (+ 596us) server_negotiation.cc:407] Connection header received
0929 02:06:01.777494 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.777498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.777613 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.777707 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:01.778334 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.781404 (+ 3070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.783660 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.784347 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.784516 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.785066 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:01.785215 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:01.785318 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.785486 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.785577 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:06:01.808697 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.801924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:52413 (local address 127.11.182.195:40173)
0929 02:06:01.802117 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:01.802123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:01.802597 (+ 474us) server_negotiation.cc:407] Connection header received
0929 02:06:01.802644 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:01.802649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:01.802710 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:01.803145 (+ 435us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:01.803252 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.804375 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.806329 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.807050 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.807284 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.807812 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:01.807963 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:01.808099 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:01.808250 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 02:06:01.808359 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":72}
I20260929 02:06:01.808763 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:01.802172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:40173 (local address 127.11.182.195:52413)
0929 02:06:01.802339 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:01.802357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:01.802492 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:01.802795 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:01.802799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:01.802809 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:06:01.803035 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.803045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.804531 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.804537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.806196 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:01.806210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:01.807277 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:01.807283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:01.807523 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:01.807598 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:06:01.808112 (+ 514us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:06:01.808123 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:06:01.808192 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 02:06:01.808347 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":23}
I20260929 02:06:01.819540 13914 raft_consensus.cc:1261] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Refusing update from remote peer 48df04fc8dc641e49c18563a9ff7fa15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:01.819557 14058 raft_consensus.cc:1261] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Refusing update from remote peer 48df04fc8dc641e49c18563a9ff7fa15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:01.820295 14299 consensus_queue.cc:1035] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:01.820750 14269 consensus_queue.cc:1035] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:01.829399 14288 mvcc.cc:205] Tried to move back new op lower bound from 7334492413207187456 to 7334492412225785856. Current Snapshot: MvccSnapshot[applied={T|T < 7334492413207187456}]
I20260929 02:06:01.830714 14289 mvcc.cc:205] Tried to move back new op lower bound from 7334492413207187456 to 7334492412225785856. Current Snapshot: MvccSnapshot[applied={T|T < 7334492413207187456}]
I20260929 02:06:01.893731 13756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:01.894001 13756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:01.897917 13756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff: Sending DeleteTablet for 3 replicas of tablet a6c6a33774564cd18888770bd2ed7146
I20260929 02:06:01.899384 14174 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c6a33774564cd18888770bd2ed7146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:01 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:36226
I20260929 02:06:01.899780 13894 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c6a33774564cd18888770bd2ed7146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:01 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:49526
I20260929 02:06:01.900146 14034 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c6a33774564cd18888770bd2ed7146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:01 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:42730
I20260929 02:06:01.901655 13757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:01.902463 13757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:01.906836 14306 tablet_replica.cc:330] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: stopping tablet replica
I20260929 02:06:01.907485 14306 raft_consensus.cc:2229] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:01.907941 14306 raft_consensus.cc:2258] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:01.910132 14305 tablet_replica.cc:330] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: stopping tablet replica
I20260929 02:06:01.910400 14304 tablet_replica.cc:330] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: stopping tablet replica
I20260929 02:06:01.911051 14305 raft_consensus.cc:2229] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:01.911499 14305 raft_consensus.cc:2258] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:01.912941 14033 tablet_service.cc:1467] Processing CreateTablet for tablet e342d9c664b34756b3249fe8fea9f3a8 (DEFAULT_TABLE table=test-table [id=0d9b632440be48a880b4384382b2add6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:01.912930 14174 tablet_service.cc:1467] Processing CreateTablet for tablet e342d9c664b34756b3249fe8fea9f3a8 (DEFAULT_TABLE table=test-table [id=0d9b632440be48a880b4384382b2add6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:01.913426 14033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e342d9c664b34756b3249fe8fea9f3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:01.913443 14174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e342d9c664b34756b3249fe8fea9f3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:01.914307 13894 tablet_service.cc:1467] Processing CreateTablet for tablet e342d9c664b34756b3249fe8fea9f3a8 (DEFAULT_TABLE table=test-table [id=0d9b632440be48a880b4384382b2add6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:01.914747 13894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e342d9c664b34756b3249fe8fea9f3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:01.920414 14265 tablet_bootstrap.cc:492] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: Bootstrap starting.
I20260929 02:06:01.920658 14305 ts_tablet_manager.cc:1905] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:01.922199 14265 tablet_bootstrap.cc:654] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:01.925978 14264 tablet_bootstrap.cc:492] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: Bootstrap starting.
I20260929 02:06:01.926028 14263 tablet_bootstrap.cc:492] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: Bootstrap starting.
I20260929 02:06:01.927416 14304 raft_consensus.cc:2229] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:01.927807 14264 tablet_bootstrap.cc:654] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:01.928145 14263 tablet_bootstrap.cc:654] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:01.928802 14304 raft_consensus.cc:2258] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:01.930045 14306 ts_tablet_manager.cc:1905] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:01.936434 14304 ts_tablet_manager.cc:1905] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:01.939536 14305 ts_tablet_manager.cc:1918] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:01.939823 14305 log.cc:1196] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/a6c6a33774564cd18888770bd2ed7146
I20260929 02:06:01.940433 14305 ts_tablet_manager.cc:1939] T a6c6a33774564cd18888770bd2ed7146 P 90862fe507bb4dbd800ff84e869c3914: Deleting consensus metadata
I20260929 02:06:01.940726 14264 tablet_bootstrap.cc:492] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: No bootstrap required, opened a new log
I20260929 02:06:01.941015 14264 ts_tablet_manager.cc:1397] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260929 02:06:01.941154 14265 tablet_bootstrap.cc:492] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: No bootstrap required, opened a new log
I20260929 02:06:01.941359 14265 ts_tablet_manager.cc:1397] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.003s
I20260929 02:06:01.941962 14264 raft_consensus.cc:348] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.942306 14264 raft_consensus.cc:374] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:01.942443 14264 raft_consensus.cc:729] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90862fe507bb4dbd800ff84e869c3914, State: Initialized, Role: FOLLOWER
I20260929 02:06:01.942395 13744 catalog_manager.cc:4841] TS 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251): tablet a6c6a33774564cd18888770bd2ed7146 (table test-table [id=dce844efc16a4987927172661830f367]) successfully deleted
I20260929 02:06:01.942092 14265 raft_consensus.cc:348] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.942817 14264 consensus_queue.cc:260] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.943024 14265 raft_consensus.cc:374] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:01.943212 14265 raft_consensus.cc:729] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee20f6423bac4a23876e09f27c1f5b8b, State: Initialized, Role: FOLLOWER
I20260929 02:06:01.943611 14265 consensus_queue.cc:260] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.944530 14265 ts_tablet_manager.cc:1428] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:01.946799 14263 tablet_bootstrap.cc:492] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: No bootstrap required, opened a new log
I20260929 02:06:01.947088 14263 ts_tablet_manager.cc:1397] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20260929 02:06:01.948071 14263 raft_consensus.cc:348] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.948390 14263 raft_consensus.cc:374] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:01.948545 14263 raft_consensus.cc:729] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48df04fc8dc641e49c18563a9ff7fa15, State: Initialized, Role: FOLLOWER
I20260929 02:06:01.948884 14263 consensus_queue.cc:260] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:01.950578 14263 ts_tablet_manager.cc:1428] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:01.951231 14264 ts_tablet_manager.cc:1428] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260929 02:06:01.953576 14306 ts_tablet_manager.cc:1918] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:01.953789 14306 log.cc:1196] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/a6c6a33774564cd18888770bd2ed7146
I20260929 02:06:01.954314 14306 ts_tablet_manager.cc:1939] T a6c6a33774564cd18888770bd2ed7146 P ee20f6423bac4a23876e09f27c1f5b8b: Deleting consensus metadata
I20260929 02:06:01.956306 13744 catalog_manager.cc:4841] TS ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007): tablet a6c6a33774564cd18888770bd2ed7146 (table test-table [id=dce844efc16a4987927172661830f367]) successfully deleted
I20260929 02:06:01.957772 14304 ts_tablet_manager.cc:1918] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:01.957967 14304 log.cc:1196] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/a6c6a33774564cd18888770bd2ed7146
I20260929 02:06:01.958451 14304 ts_tablet_manager.cc:1939] T a6c6a33774564cd18888770bd2ed7146 P 48df04fc8dc641e49c18563a9ff7fa15: Deleting consensus metadata
I20260929 02:06:01.960130 13744 catalog_manager.cc:4841] TS 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195:40173): tablet a6c6a33774564cd18888770bd2ed7146 (table test-table [id=dce844efc16a4987927172661830f367]) successfully deleted
I20260929 02:06:02.068835 14273 raft_consensus.cc:482] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:02.069099 14273 raft_consensus.cc:504] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:02.069799 14273 leader_election.cc:288] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251), ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007)
I20260929 02:06:02.070418 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e342d9c664b34756b3249fe8fea9f3a8" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90862fe507bb4dbd800ff84e869c3914" is_pre_election: true
I20260929 02:06:02.070504 14058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e342d9c664b34756b3249fe8fea9f3a8" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" is_pre_election: true
I20260929 02:06:02.070662 13914 raft_consensus.cc:2454] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 0.
I20260929 02:06:02.070930 14058 raft_consensus.cc:2454] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 0.
I20260929 02:06:02.071301 14127 leader_election.cc:302] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48df04fc8dc641e49c18563a9ff7fa15, 90862fe507bb4dbd800ff84e869c3914; no voters:
I20260929 02:06:02.071637 14273 raft_consensus.cc:2781] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:02.071754 14273 raft_consensus.cc:482] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:02.071877 14273 raft_consensus.cc:3037] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:02.074465 14273 raft_consensus.cc:504] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:02.075206 14273 leader_election.cc:288] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 election: Requested vote from peers 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251), ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007)
I20260929 02:06:02.075804 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e342d9c664b34756b3249fe8fea9f3a8" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90862fe507bb4dbd800ff84e869c3914"
I20260929 02:06:02.075910 14058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e342d9c664b34756b3249fe8fea9f3a8" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b"
I20260929 02:06:02.076026 13914 raft_consensus.cc:3037] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:02.076174 14058 raft_consensus.cc:3037] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:02.078154 13914 raft_consensus.cc:2454] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 1.
I20260929 02:06:02.078666 14127 leader_election.cc:302] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48df04fc8dc641e49c18563a9ff7fa15, 90862fe507bb4dbd800ff84e869c3914; no voters:
I20260929 02:06:02.079015 14273 raft_consensus.cc:2781] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:02.079236 14273 raft_consensus.cc:686] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 LEADER]: Becoming Leader. State: Replica: 48df04fc8dc641e49c18563a9ff7fa15, State: Running, Role: LEADER
I20260929 02:06:02.079401 14058 raft_consensus.cc:2454] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 1.
I20260929 02:06:02.079581 14273 consensus_queue.cc:237] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:02.082919 13756 catalog_manager.cc:5495] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195). New cstate: current_term: 1 leader_uuid: "48df04fc8dc641e49c18563a9ff7fa15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } health_report { overall_health: HEALTHY } } }
I20260929 02:06:02.143729 14058 raft_consensus.cc:1261] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Refusing update from remote peer 48df04fc8dc641e49c18563a9ff7fa15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:02.143915 13914 raft_consensus.cc:1261] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Refusing update from remote peer 48df04fc8dc641e49c18563a9ff7fa15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:02.144385 14273 consensus_queue.cc:1035] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:02.144650 14299 consensus_queue.cc:1035] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:06:02.178483 14288 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:06:03.779495 13756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:03.779706 13756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:03.784132 13756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff: Sending DeleteTablet for 3 replicas of tablet e342d9c664b34756b3249fe8fea9f3a8
I20260929 02:06:03.785178 13894 tablet_service.cc:1514] Processing DeleteTablet for tablet e342d9c664b34756b3249fe8fea9f3a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:03 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:49526
I20260929 02:06:03.785748 14033 tablet_service.cc:1514] Processing DeleteTablet for tablet e342d9c664b34756b3249fe8fea9f3a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:03 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:42730
I20260929 02:06:03.787810 14174 tablet_service.cc:1514] Processing DeleteTablet for tablet e342d9c664b34756b3249fe8fea9f3a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:03 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:36226
I20260929 02:06:03.799712 14334 tablet_replica.cc:330] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: stopping tablet replica
I20260929 02:06:03.800640 14333 tablet_replica.cc:330] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: stopping tablet replica
I20260929 02:06:03.800859 14333 raft_consensus.cc:2229] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:03.801108 14334 raft_consensus.cc:2229] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:03.801489 14333 raft_consensus.cc:2258] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:03.801918 14334 raft_consensus.cc:2258] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:03.802076 14332 tablet_replica.cc:330] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: stopping tablet replica
I20260929 02:06:03.802280 14332 raft_consensus.cc:2229] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:03.802539 14332 raft_consensus.cc:2258] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:03.802700 14333 ts_tablet_manager.cc:1905] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:03.803550 14332 ts_tablet_manager.cc:1905] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:03.804596 14334 ts_tablet_manager.cc:1905] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:03.811641 14333 ts_tablet_manager.cc:1918] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:06:03.812109 14332 ts_tablet_manager.cc:1918] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:06:03.812317 14332 log.cc:1196] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/e342d9c664b34756b3249fe8fea9f3a8
I20260929 02:06:03.812213 14333 log.cc:1196] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/e342d9c664b34756b3249fe8fea9f3a8
I20260929 02:06:03.812989 14333 ts_tablet_manager.cc:1939] T e342d9c664b34756b3249fe8fea9f3a8 P ee20f6423bac4a23876e09f27c1f5b8b: Deleting consensus metadata
I20260929 02:06:03.813172 14332 ts_tablet_manager.cc:1939] T e342d9c664b34756b3249fe8fea9f3a8 P 90862fe507bb4dbd800ff84e869c3914: Deleting consensus metadata
I20260929 02:06:03.814589 13744 catalog_manager.cc:4841] TS ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007): tablet e342d9c664b34756b3249fe8fea9f3a8 (table test-table [id=0d9b632440be48a880b4384382b2add6]) successfully deleted
I20260929 02:06:03.814962 13744 catalog_manager.cc:4841] TS 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251): tablet e342d9c664b34756b3249fe8fea9f3a8 (table test-table [id=0d9b632440be48a880b4384382b2add6]) successfully deleted
I20260929 02:06:03.817830 14334 ts_tablet_manager.cc:1918] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:06:03.817981 14334 log.cc:1196] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/e342d9c664b34756b3249fe8fea9f3a8
I20260929 02:06:03.818418 14334 ts_tablet_manager.cc:1939] T e342d9c664b34756b3249fe8fea9f3a8 P 48df04fc8dc641e49c18563a9ff7fa15: Deleting consensus metadata
I20260929 02:06:03.819597 13744 catalog_manager.cc:4841] TS 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195:40173): tablet e342d9c664b34756b3249fe8fea9f3a8 (table test-table [id=0d9b632440be48a880b4384382b2add6]) successfully deleted
Sep 29 02:06:03 dist-test-slave-q06p krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:06:03 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:03.855726 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:03.840174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.11.182.254:45287)
0929 02:06:03.840777 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:03.840782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:03.840869 (+ 87us) server_negotiation.cc:407] Connection header received
0929 02:06:03.840926 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:03.840930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:03.840989 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:03.841077 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:03.842472 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.843343 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.844723 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.845147 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.845262 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:03.849712 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:03.849718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:03.849721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:03.849759 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:03.852766 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:03.853537 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:03.853542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:03.853544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:03.853682 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:03.854151 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:03.854155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:03.854157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:03.854514 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:03.854699 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:03.855246 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 02:06:03.855449 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":330,"threads_started":1}
I20260929 02:06:03.859550 13756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:03.860260 13756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:03.869652 14174 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1b1dd45d964c9da6ea7be9eb064eea (DEFAULT_TABLE table=test-table [id=4ab010dc7974497ea473e32b25ec3eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:03.869617 14033 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1b1dd45d964c9da6ea7be9eb064eea (DEFAULT_TABLE table=test-table [id=4ab010dc7974497ea473e32b25ec3eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:03.869648 13894 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1b1dd45d964c9da6ea7be9eb064eea (DEFAULT_TABLE table=test-table [id=4ab010dc7974497ea473e32b25ec3eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:03.870124 13894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1b1dd45d964c9da6ea7be9eb064eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:03.870121 14033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1b1dd45d964c9da6ea7be9eb064eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:03.870137 14174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1b1dd45d964c9da6ea7be9eb064eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:03.875854 14343 tablet_bootstrap.cc:492] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: Bootstrap starting.
I20260929 02:06:03.878055 14343 tablet_bootstrap.cc:654] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:03.879763 14344 tablet_bootstrap.cc:492] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: Bootstrap starting.
I20260929 02:06:03.881488 14345 tablet_bootstrap.cc:492] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: Bootstrap starting.
I20260929 02:06:03.882817 14343 tablet_bootstrap.cc:492] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: No bootstrap required, opened a new log
I20260929 02:06:03.883064 14343 ts_tablet_manager.cc:1397] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260929 02:06:03.883612 14345 tablet_bootstrap.cc:654] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:03.883989 14343 raft_consensus.cc:348] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.884271 14343 raft_consensus.cc:374] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:03.884372 14343 raft_consensus.cc:729] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90862fe507bb4dbd800ff84e869c3914, State: Initialized, Role: FOLLOWER
I20260929 02:06:03.884644 14343 consensus_queue.cc:260] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.886801 14344 tablet_bootstrap.cc:654] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:03.888401 14343 ts_tablet_manager.cc:1428] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260929 02:06:03.889894 14345 tablet_bootstrap.cc:492] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: No bootstrap required, opened a new log
I20260929 02:06:03.890074 14345 ts_tablet_manager.cc:1397] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 02:06:03.890355 14344 tablet_bootstrap.cc:492] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: No bootstrap required, opened a new log
I20260929 02:06:03.890544 14344 ts_tablet_manager.cc:1397] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20260929 02:06:03.890978 14345 raft_consensus.cc:348] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.891268 14345 raft_consensus.cc:374] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:03.891388 14345 raft_consensus.cc:729] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee20f6423bac4a23876e09f27c1f5b8b, State: Initialized, Role: FOLLOWER
I20260929 02:06:03.891394 14344 raft_consensus.cc:348] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.891644 14344 raft_consensus.cc:374] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:03.891758 14344 raft_consensus.cc:729] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48df04fc8dc641e49c18563a9ff7fa15, State: Initialized, Role: FOLLOWER
I20260929 02:06:03.891645 14345 consensus_queue.cc:260] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.892014 14344 consensus_queue.cc:260] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.892639 14344 ts_tablet_manager.cc:1428] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:06:03.893213 14345 ts_tablet_manager.cc:1428] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:03.930616 14348 raft_consensus.cc:482] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:03.930955 14348 raft_consensus.cc:504] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.932730 14348 leader_election.cc:288] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007), 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195:40173)
I20260929 02:06:03.944375 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:03.933197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:44571 (local address 127.11.182.194:40007)
0929 02:06:03.934030 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:03.934040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:03.934420 (+ 380us) server_negotiation.cc:407] Connection header received
0929 02:06:03.934688 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:03.934695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:03.934812 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:03.936250 (+ 1438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:03.936507 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.937893 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.941006 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.942782 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.943198 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:03.943285 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:03.943778 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 02:06:03.943962 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":383,"threads_started":1}
I20260929 02:06:03.944729 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:03.933036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:40007 (local address 127.11.182.193:44571)
0929 02:06:03.934305 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:03.934322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:03.934506 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:03.935639 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:03.935646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:03.935679 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:03.936015 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:03.936030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.938104 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.938108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:03.940757 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:03.940780 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.943226 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.943235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:03.943525 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:03.943569 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 02:06:03.943684 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 02:06:03.943900 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":282,"thread_start_us":760,"threads_started":1}
I20260929 02:06:03.944738 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:03.934513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:40173 (local address 127.11.182.193:39967)
0929 02:06:03.935341 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:03.935371 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:03.935590 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:03.936044 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:03.936050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:03.936086 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:03.936519 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:03.936531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.938102 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.938108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:03.940758 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:03.940780 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.943240 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.943245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:03.943499 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:03.943567 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 02:06:03.943684 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 02:06:03.943900 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":146,"thread_start_us":381,"threads_started":1}
I20260929 02:06:03.945379 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:03.934608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:39967 (local address 127.11.182.195:40173)
0929 02:06:03.935663 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:03.935669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:03.935722 (+ 53us) server_negotiation.cc:407] Connection header received
0929 02:06:03.935797 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:03.935804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:03.935881 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:03.935992 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:03.936722 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.937933 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.940998 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.942998 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.943211 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:03.943285 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:03.944790 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0929 02:06:03.944959 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":773,"threads_started":1}
I20260929 02:06:03.945938 14058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1dd45d964c9da6ea7be9eb064eea" candidate_uuid: "90862fe507bb4dbd800ff84e869c3914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" is_pre_election: true
I20260929 02:06:03.946305 14058 raft_consensus.cc:2454] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90862fe507bb4dbd800ff84e869c3914 in term 0.
I20260929 02:06:03.946518 14194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1dd45d964c9da6ea7be9eb064eea" candidate_uuid: "90862fe507bb4dbd800ff84e869c3914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48df04fc8dc641e49c18563a9ff7fa15" is_pre_election: true
I20260929 02:06:03.946758 14269 raft_consensus.cc:482] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:03.946995 14269 raft_consensus.cc:504] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.947037 13847 leader_election.cc:302] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90862fe507bb4dbd800ff84e869c3914, ee20f6423bac4a23876e09f27c1f5b8b; no voters:
I20260929 02:06:03.947427 14194 raft_consensus.cc:2454] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90862fe507bb4dbd800ff84e869c3914 in term 0.
I20260929 02:06:03.947540 14348 raft_consensus.cc:2781] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:03.947692 14348 raft_consensus.cc:482] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:03.947746 14269 leader_election.cc:288] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251), ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007)
I20260929 02:06:03.947885 14348 raft_consensus.cc:3037] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:03.948357 14058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1dd45d964c9da6ea7be9eb064eea" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" is_pre_election: true
I20260929 02:06:03.948604 14058 raft_consensus.cc:2454] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48df04fc8dc641e49c18563a9ff7fa15 in term 0.
I20260929 02:06:03.948916 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1dd45d964c9da6ea7be9eb064eea" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90862fe507bb4dbd800ff84e869c3914" is_pre_election: true
I20260929 02:06:03.949086 14127 leader_election.cc:302] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48df04fc8dc641e49c18563a9ff7fa15, ee20f6423bac4a23876e09f27c1f5b8b; no voters:
I20260929 02:06:03.949368 14269 raft_consensus.cc:2781] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:03.949506 14269 raft_consensus.cc:482] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:03.949589 14269 raft_consensus.cc:3037] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:03.952020 14348 raft_consensus.cc:504] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.952771 13914 raft_consensus.cc:2379] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48df04fc8dc641e49c18563a9ff7fa15 in current term 1: Already voted for candidate 90862fe507bb4dbd800ff84e869c3914 in this term.
I20260929 02:06:03.952775 14269 raft_consensus.cc:504] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.953550 14058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1dd45d964c9da6ea7be9eb064eea" candidate_uuid: "90862fe507bb4dbd800ff84e869c3914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b"
I20260929 02:06:03.953729 14269 leader_election.cc:288] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 election: Requested vote from peers 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251), ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007)
I20260929 02:06:03.953827 14058 raft_consensus.cc:3037] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:03.954095 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1dd45d964c9da6ea7be9eb064eea" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90862fe507bb4dbd800ff84e869c3914"
I20260929 02:06:03.954216 14194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1dd45d964c9da6ea7be9eb064eea" candidate_uuid: "90862fe507bb4dbd800ff84e869c3914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48df04fc8dc641e49c18563a9ff7fa15"
I20260929 02:06:03.954365 13914 raft_consensus.cc:2379] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 48df04fc8dc641e49c18563a9ff7fa15 in current term 1: Already voted for candidate 90862fe507bb4dbd800ff84e869c3914 in this term.
I20260929 02:06:03.954556 14194 raft_consensus.cc:2379] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 90862fe507bb4dbd800ff84e869c3914 in current term 1: Already voted for candidate 48df04fc8dc641e49c18563a9ff7fa15 in this term.
I20260929 02:06:03.954591 14348 leader_election.cc:288] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [CANDIDATE]: Term 1 election: Requested vote from peers ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007), 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195:40173)
I20260929 02:06:03.956256 14058 raft_consensus.cc:2454] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90862fe507bb4dbd800ff84e869c3914 in term 1.
I20260929 02:06:03.956601 14059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1dd45d964c9da6ea7be9eb064eea" candidate_uuid: "48df04fc8dc641e49c18563a9ff7fa15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b"
I20260929 02:06:03.956898 14059 raft_consensus.cc:2379] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 48df04fc8dc641e49c18563a9ff7fa15 in current term 1: Already voted for candidate 90862fe507bb4dbd800ff84e869c3914 in this term.
I20260929 02:06:03.957193 13847 leader_election.cc:302] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90862fe507bb4dbd800ff84e869c3914, ee20f6423bac4a23876e09f27c1f5b8b; no voters: 48df04fc8dc641e49c18563a9ff7fa15
I20260929 02:06:03.957310 14127 leader_election.cc:302] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48df04fc8dc641e49c18563a9ff7fa15; no voters: 90862fe507bb4dbd800ff84e869c3914, ee20f6423bac4a23876e09f27c1f5b8b
I20260929 02:06:03.957625 14348 raft_consensus.cc:2781] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:03.957751 14269 raft_consensus.cc:2726] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 02:06:03.958244 14348 raft_consensus.cc:686] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 1 LEADER]: Becoming Leader. State: Replica: 90862fe507bb4dbd800ff84e869c3914, State: Running, Role: LEADER
I20260929 02:06:03.958657 14348 consensus_queue.cc:237] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:03.962215 13756 catalog_manager.cc:5495] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193). New cstate: current_term: 1 leader_uuid: "90862fe507bb4dbd800ff84e869c3914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } health_report { overall_health: UNKNOWN } } }
I20260929 02:06:04.001991 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:03.995351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.11.182.193:38251)
0929 02:06:03.996229 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:03.996239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:03.996261 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:06:03.996374 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:03.996378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:03.996455 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:03.996667 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:03.997157 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.997988 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:03.999488 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:03.999965 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:04.000127 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:04.000677 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:04.001340 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:04.001477 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:04.001627 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 02:06:04.001710 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":608,"threads_started":1}
I20260929 02:06:04.021420 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:04.015800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:43033 (local address 127.11.182.195:40173)
0929 02:06:04.016143 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:04.016149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:04.016489 (+ 340us) server_negotiation.cc:407] Connection header received
0929 02:06:04.016631 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:04.016635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:04.016699 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:04.016781 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:04.017260 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:04.017990 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:04.019620 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:04.020074 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:04.020183 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:04.020699 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:04.020820 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:04.020902 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:04.021108 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 02:06:04.021183 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260929 02:06:04.021476 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:04.015560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:40173 (local address 127.11.182.193:43033)
0929 02:06:04.016365 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:04.016381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:04.016516 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:04.016822 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:04.016826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:04.016836 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:06:04.017100 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:04.017108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:04.018134 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:04.018138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:04.019404 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:04.019439 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:04.020297 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:04.020301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:04.020423 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:04.020486 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:06:04.020938 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:06:04.020959 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 02:06:04.021031 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 02:06:04.021143 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":419,"threads_started":1}
I20260929 02:06:04.032577 14059 raft_consensus.cc:1261] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Refusing update from remote peer 90862fe507bb4dbd800ff84e869c3914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:04.033003 14194 raft_consensus.cc:1261] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Refusing update from remote peer 90862fe507bb4dbd800ff84e869c3914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:04.033659 14355 consensus_queue.cc:1035] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:04.034075 14348 consensus_queue.cc:1035] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:04.075101 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:04.068225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:48421 (local address 127.11.182.193:38251)
0929 02:06:04.068441 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:04.068448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:04.069326 (+ 878us) server_negotiation.cc:407] Connection header received
0929 02:06:04.069553 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:04.069559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:04.070012 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:04.070171 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:04.070908 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:04.071694 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:04.073143 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:04.073711 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:04.073857 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:04.074376 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:04.074479 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:04.074581 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:04.074692 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 02:06:04.074772 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:06:04.075248 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:04.068024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38251 (local address 127.11.182.195:48421)
0929 02:06:04.069204 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:04.069225 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:04.069389 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:04.070183 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:04.070189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:04.070204 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:06:04.070697 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:04.070708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:04.071866 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:04.071872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:04.072981 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:04.072994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:04.073937 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:04.073944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:04.074160 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:04.074207 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:06:04.074581 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:06:04.074595 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 02:06:04.074675 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 02:06:04.074814 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":302,"threads_started":1}
I20260929 02:06:04.108500 13755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:04.108714 13755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:04.112450 13755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff: Sending DeleteTablet for 3 replicas of tablet 5b1b1dd45d964c9da6ea7be9eb064eea
I20260929 02:06:04.113513 13894 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1b1dd45d964c9da6ea7be9eb064eea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:04 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:49526
I20260929 02:06:04.113883 14332 tablet_replica.cc:330] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: stopping tablet replica
I20260929 02:06:04.113816 14174 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1b1dd45d964c9da6ea7be9eb064eea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:04 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:36226
I20260929 02:06:04.114087 14332 raft_consensus.cc:2229] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:04.114454 14033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1b1dd45d964c9da6ea7be9eb064eea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:04 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:42730
I20260929 02:06:04.114667 14332 raft_consensus.cc:2258] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:04.114743 14333 tablet_replica.cc:330] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: stopping tablet replica
I20260929 02:06:04.114897 14334 tablet_replica.cc:330] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: stopping tablet replica
I20260929 02:06:04.114979 14333 raft_consensus.cc:2229] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:04.115132 14334 raft_consensus.cc:2229] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:04.115187 14333 raft_consensus.cc:2258] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:04.115343 14334 raft_consensus.cc:2258] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:04.116467 14332 ts_tablet_manager.cc:1905] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:04.116379 13755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:04.117125 13755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:04.119247 14334 ts_tablet_manager.cc:1905] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:04.124068 14333 ts_tablet_manager.cc:1905] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:04.124214 14332 ts_tablet_manager.cc:1918] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:04.124796 14332 log.cc:1196] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/5b1b1dd45d964c9da6ea7be9eb064eea
I20260929 02:06:04.125451 14332 ts_tablet_manager.cc:1939] T 5b1b1dd45d964c9da6ea7be9eb064eea P 90862fe507bb4dbd800ff84e869c3914: Deleting consensus metadata
I20260929 02:06:04.130927 14334 ts_tablet_manager.cc:1918] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:04.131189 14334 log.cc:1196] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/5b1b1dd45d964c9da6ea7be9eb064eea
I20260929 02:06:04.132122 14033 tablet_service.cc:1467] Processing CreateTablet for tablet 0c77ffca7c09438a9767b9526c8925c5 (DEFAULT_TABLE table=test-table [id=321d35f31b414df99ff054bb36557666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:04.132628 14033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c77ffca7c09438a9767b9526c8925c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:04.132862 14334 ts_tablet_manager.cc:1939] T 5b1b1dd45d964c9da6ea7be9eb064eea P 48df04fc8dc641e49c18563a9ff7fa15: Deleting consensus metadata
I20260929 02:06:04.133271 14333 ts_tablet_manager.cc:1918] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:04.134946 14174 tablet_service.cc:1467] Processing CreateTablet for tablet 0c77ffca7c09438a9767b9526c8925c5 (DEFAULT_TABLE table=test-table [id=321d35f31b414df99ff054bb36557666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:04.135715 13894 tablet_service.cc:1467] Processing CreateTablet for tablet 0c77ffca7c09438a9767b9526c8925c5 (DEFAULT_TABLE table=test-table [id=321d35f31b414df99ff054bb36557666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:04.135939 14174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c77ffca7c09438a9767b9526c8925c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:04.136176 13894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c77ffca7c09438a9767b9526c8925c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:04.137178 13744 catalog_manager.cc:4841] TS 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251): tablet 5b1b1dd45d964c9da6ea7be9eb064eea (table test-table [id=4ab010dc7974497ea473e32b25ec3eb1]) successfully deleted
I20260929 02:06:04.137424 13744 catalog_manager.cc:4841] TS 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195:40173): tablet 5b1b1dd45d964c9da6ea7be9eb064eea (table test-table [id=4ab010dc7974497ea473e32b25ec3eb1]) successfully deleted
I20260929 02:06:04.143635 14343 tablet_bootstrap.cc:492] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: Bootstrap starting.
I20260929 02:06:04.143823 14344 tablet_bootstrap.cc:492] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: Bootstrap starting.
I20260929 02:06:04.146082 14343 tablet_bootstrap.cc:654] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:04.146271 14344 tablet_bootstrap.cc:654] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:04.146811 14345 tablet_bootstrap.cc:492] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: Bootstrap starting.
I20260929 02:06:04.149749 14345 tablet_bootstrap.cc:654] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:04.134423 14333 log.cc:1196] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/5b1b1dd45d964c9da6ea7be9eb064eea
I20260929 02:06:04.153827 14333 ts_tablet_manager.cc:1939] T 5b1b1dd45d964c9da6ea7be9eb064eea P ee20f6423bac4a23876e09f27c1f5b8b: Deleting consensus metadata
I20260929 02:06:04.154178 14345 tablet_bootstrap.cc:492] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: No bootstrap required, opened a new log
I20260929 02:06:04.154388 14345 ts_tablet_manager.cc:1397] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 02:06:04.155301 13744 catalog_manager.cc:4841] TS ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007): tablet 5b1b1dd45d964c9da6ea7be9eb064eea (table test-table [id=4ab010dc7974497ea473e32b25ec3eb1]) successfully deleted
I20260929 02:06:04.155414 14345 raft_consensus.cc:348] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:04.155723 14345 raft_consensus.cc:374] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:04.155838 14345 raft_consensus.cc:729] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee20f6423bac4a23876e09f27c1f5b8b, State: Initialized, Role: FOLLOWER
I20260929 02:06:04.156190 14345 consensus_queue.cc:260] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:04.156438 14344 tablet_bootstrap.cc:492] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: No bootstrap required, opened a new log
I20260929 02:06:04.156664 14344 ts_tablet_manager.cc:1397] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260929 02:06:04.157428 14343 tablet_bootstrap.cc:492] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: No bootstrap required, opened a new log
I20260929 02:06:04.157593 14343 ts_tablet_manager.cc:1397] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20260929 02:06:04.157481 14344 raft_consensus.cc:348] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:04.158179 14344 raft_consensus.cc:374] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:04.158289 14344 raft_consensus.cc:729] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48df04fc8dc641e49c18563a9ff7fa15, State: Initialized, Role: FOLLOWER
I20260929 02:06:04.158569 14344 consensus_queue.cc:260] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:04.158818 14343 raft_consensus.cc:348] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:04.159139 14343 raft_consensus.cc:374] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:04.159210 14343 raft_consensus.cc:729] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90862fe507bb4dbd800ff84e869c3914, State: Initialized, Role: FOLLOWER
I20260929 02:06:04.159740 14345 ts_tablet_manager.cc:1428] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260929 02:06:04.159878 14344 ts_tablet_manager.cc:1428] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 02:06:04.160250 14343 consensus_queue.cc:260] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:04.160799 14343 ts_tablet_manager.cc:1428] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 02:06:04.167812 14364 raft_consensus.cc:482] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:04.168071 14364 raft_consensus.cc:504] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:04.169062 14364 leader_election.cc:288] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007), 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195:40173)
I20260929 02:06:04.169611 14194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c77ffca7c09438a9767b9526c8925c5" candidate_uuid: "90862fe507bb4dbd800ff84e869c3914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48df04fc8dc641e49c18563a9ff7fa15" is_pre_election: true
I20260929 02:06:04.169629 14059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c77ffca7c09438a9767b9526c8925c5" candidate_uuid: "90862fe507bb4dbd800ff84e869c3914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" is_pre_election: true
I20260929 02:06:04.169863 14059 raft_consensus.cc:2454] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90862fe507bb4dbd800ff84e869c3914 in term 0.
I20260929 02:06:04.169873 14194 raft_consensus.cc:2454] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90862fe507bb4dbd800ff84e869c3914 in term 0.
I20260929 02:06:04.170361 13847 leader_election.cc:302] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90862fe507bb4dbd800ff84e869c3914, ee20f6423bac4a23876e09f27c1f5b8b; no voters:
I20260929 02:06:04.170677 14364 raft_consensus.cc:2781] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:04.170807 14364 raft_consensus.cc:482] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:04.170938 14364 raft_consensus.cc:3037] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:04.174938 14364 raft_consensus.cc:504] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:04.175705 14364 leader_election.cc:288] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [CANDIDATE]: Term 1 election: Requested vote from peers ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007), 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195:40173)
I20260929 02:06:04.176360 14194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c77ffca7c09438a9767b9526c8925c5" candidate_uuid: "90862fe507bb4dbd800ff84e869c3914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48df04fc8dc641e49c18563a9ff7fa15"
I20260929 02:06:04.176347 14059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c77ffca7c09438a9767b9526c8925c5" candidate_uuid: "90862fe507bb4dbd800ff84e869c3914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee20f6423bac4a23876e09f27c1f5b8b"
I20260929 02:06:04.176602 14059 raft_consensus.cc:3037] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:04.176633 14194 raft_consensus.cc:3037] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:04.178679 14194 raft_consensus.cc:2454] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90862fe507bb4dbd800ff84e869c3914 in term 1.
I20260929 02:06:04.179294 13847 leader_election.cc:302] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48df04fc8dc641e49c18563a9ff7fa15, 90862fe507bb4dbd800ff84e869c3914; no voters:
I20260929 02:06:04.179615 14059 raft_consensus.cc:2454] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90862fe507bb4dbd800ff84e869c3914 in term 1.
I20260929 02:06:04.179723 14364 raft_consensus.cc:2781] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:04.181488 14364 raft_consensus.cc:686] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 1 LEADER]: Becoming Leader. State: Replica: 90862fe507bb4dbd800ff84e869c3914, State: Running, Role: LEADER
I20260929 02:06:04.181939 14364 consensus_queue.cc:237] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } }
I20260929 02:06:04.186124 13757 catalog_manager.cc:5495] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193). New cstate: current_term: 1 leader_uuid: "90862fe507bb4dbd800ff84e869c3914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90862fe507bb4dbd800ff84e869c3914" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 } health_report { overall_health: UNKNOWN } } }
I20260929 02:06:04.224634 14194 raft_consensus.cc:1261] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Refusing update from remote peer 90862fe507bb4dbd800ff84e869c3914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:04.225351 14059 raft_consensus.cc:1261] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Refusing update from remote peer 90862fe507bb4dbd800ff84e869c3914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:04.226384 14364 consensus_queue.cc:1035] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48df04fc8dc641e49c18563a9ff7fa15" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 02:06:04.226773 14355 consensus_queue.cc:1035] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee20f6423bac4a23876e09f27c1f5b8b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:06:04.263306 14317 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:06:05.868227 13757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:05.868475 13757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:05.873315 13757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e62c97e444a946f882182c0d188decff: Sending DeleteTablet for 3 replicas of tablet 0c77ffca7c09438a9767b9526c8925c5
I20260929 02:06:05.874333 13894 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c77ffca7c09438a9767b9526c8925c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:05 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:49526
I20260929 02:06:05.874401 14033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c77ffca7c09438a9767b9526c8925c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:05 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:42730
I20260929 02:06:05.874428 14173 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c77ffca7c09438a9767b9526c8925c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:05 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:36226
I20260929 02:06:05.875025 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13828
I20260929 02:06:05.887233 14400 tablet_replica.cc:330] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: stopping tablet replica
I20260929 02:06:05.887504 14399 tablet_replica.cc:330] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: stopping tablet replica
I20260929 02:06:05.887717 14399 raft_consensus.cc:2229] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:05.887720 14400 raft_consensus.cc:2229] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:05.888022 14400 raft_consensus.cc:2258] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:05.888021 14399 raft_consensus.cc:2258] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:05.889219 14400 ts_tablet_manager.cc:1905] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:05.889367 14399 ts_tablet_manager.cc:1905] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:05.896533 14398 tablet_replica.cc:330] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: stopping tablet replica
I20260929 02:06:05.896775 14398 raft_consensus.cc:2229] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:05.897179 14400 ts_tablet_manager.cc:1918] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:06:05.897197 14399 ts_tablet_manager.cc:1918] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:06:05.897220 14398 raft_consensus.cc:2258] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:05.897388 14399 log.cc:1196] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/0c77ffca7c09438a9767b9526c8925c5
I20260929 02:06:05.897424 14400 log.cc:1196] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/0c77ffca7c09438a9767b9526c8925c5
I20260929 02:06:05.897930 14399 ts_tablet_manager.cc:1939] T 0c77ffca7c09438a9767b9526c8925c5 P ee20f6423bac4a23876e09f27c1f5b8b: Deleting consensus metadata
I20260929 02:06:05.897940 14400 ts_tablet_manager.cc:1939] T 0c77ffca7c09438a9767b9526c8925c5 P 48df04fc8dc641e49c18563a9ff7fa15: Deleting consensus metadata
I20260929 02:06:05.898319 14398 ts_tablet_manager.cc:1905] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:05.899262 13744 catalog_manager.cc:4841] TS 48df04fc8dc641e49c18563a9ff7fa15 (127.11.182.195:40173): tablet 0c77ffca7c09438a9767b9526c8925c5 (table test-table [id=321d35f31b414df99ff054bb36557666]) successfully deleted
I20260929 02:06:05.899685 13744 catalog_manager.cc:4841] TS ee20f6423bac4a23876e09f27c1f5b8b (127.11.182.194:40007): tablet 0c77ffca7c09438a9767b9526c8925c5 (table test-table [id=321d35f31b414df99ff054bb36557666]) successfully deleted
I20260929 02:06:05.904739 14398 ts_tablet_manager.cc:1918] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:06:05.905239 14398 log.cc:1196] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/0c77ffca7c09438a9767b9526c8925c5
I20260929 02:06:05.905928 14398 ts_tablet_manager.cc:1939] T 0c77ffca7c09438a9767b9526c8925c5 P 90862fe507bb4dbd800ff84e869c3914: Deleting consensus metadata
Sep 29 02:06:05 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:06:05.907102 13744 catalog_manager.cc:4841] TS 90862fe507bb4dbd800ff84e869c3914 (127.11.182.193:38251): tablet 0c77ffca7c09438a9767b9526c8925c5 (table test-table [id=321d35f31b414df99ff054bb36557666]) successfully deleted
I20260929 02:06:05.916112 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:05.876982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.11.182.193:38251)
0929 02:06:05.895685 (+ 18703us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:05.895694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:05.895728 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:06:05.895807 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:05.895815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:05.895925 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:05.896052 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:05.898144 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:05.899493 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:05.901229 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:05.901870 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:05.902025 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:05.908129 (+ 6104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:05.908147 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:05.908170 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:05.908216 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:05.911613 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:05.912964 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:05.912984 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:05.912987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:05.913346 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:05.914273 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:05.914283 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:05.914287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:05.914822 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:05.915194 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:05.915511 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 02:06:05.915702 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18522,"thread_start_us":18512,"threads_started":1}
W20260929 02:06:05.916966 13959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37170
W20260929 02:06:05.917603 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13828: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:06:05.917837 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 13828
I20260929 02:06:05.942122 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13968
Sep 29 02:06:05 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:06:05.960191 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:05.943057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.11.182.194:40007)
0929 02:06:05.943602 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:05.943607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:05.943623 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:06:05.943677 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:05.943681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:05.943750 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:05.943851 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:05.945378 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:05.946155 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:05.947589 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:05.948024 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:05.948154 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:05.953310 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:05.953329 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:05.953346 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:05.953392 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:05.956617 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:05.957483 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:05.957493 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:05.957495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:05.957678 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:05.958273 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:05.958277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:05.958279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:05.958671 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:05.959258 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:05.959654 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 02:06:05.959854 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":278,"threads_started":1}
W20260929 02:06:05.961229 14099 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35484
W20260929 02:06:05.962044 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13968: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:06:05.962252 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 13968
I20260929 02:06:05.981123 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14108
Sep 29 02:06:05 dist-test-slave-q06p krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:06:05.999360 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:05.982209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.11.182.195:40173)
0929 02:06:05.982894 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:05.982901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:05.982925 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:06:05.982980 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:05.982983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:05.983043 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:05.983139 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:05.984655 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:05.985427 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:05.986834 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:05.987420 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:05.987530 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:05.992720 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:05.992738 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:05.992763 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:05.992819 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:05.996026 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:05.997057 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:05.997066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:05.997068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:05.997236 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:05.997746 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:05.997750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:05.997752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:05.998144 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:05.998411 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:05.998798 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 02:06:05.999068 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":397,"threads_started":1}
W20260929 02:06:06.000134 14239 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49726
W20260929 02:06:06.000986 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14108: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:06:06.001250 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 14108
I20260929 02:06:06.029932 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 13725
I20260929 02:06:06.031545 13797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:06.157058 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 13725
2026-09-29T02:06:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8560 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:06 dist-test-slave-q06p krb5kdc[14408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:06 dist-test-slave-q06p krb5kdc[14408](info): set up 2 sockets
Sep 29 02:06:06 dist-test-slave-q06p krb5kdc[14408](info): commencing operation
krb5kdc: starting...
W20260929 02:06:07.320039 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:07 dist-test-slave-q06p krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:06:07.530393 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41245
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:07.863107 14421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:07.863421 14421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:07.863511 14421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:07.871083 14421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:07.871208 14421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:07.871254 14421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:07.871304 14421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:07.871349 14421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:07.881742 14421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:07.883126 14421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:07.884414 14421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 02:06:07.898298 14421 server_base.cc:1034] running on GCE node
W20260929 02:06:07.898365 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:07.899132 14427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:07.899152 14429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:07.900331 14421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:07.901957 14421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:07.903209 14421 hybrid_clock.cc:648] HybridClock initialized: now 1790647567903156 us; error 31 us; skew 500 ppm
I20260929 02:06:07.903651 14421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:07 dist-test-slave-q06p krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:07.907670 14421 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:06:07.909497 14421 webserver.cc:460] Webserver started at http://127.11.182.254:45011/ using document root <none> and password file <none>
I20260929 02:06:07.910285 14421 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:07.910447 14421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:07.910784 14421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:07.914186 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "7633a7168c894b6396013d438f669589"
format_stamp: "Formatted at 2026-09-29 02:06:07 on dist-test-slave-q06p"
server_key: "a010635c2cdf5650bfca190a5a748174"
server_key_iv: "fc85674ace66b05df0f4e6af99a2dc95"
server_key_version: "encryptionkey@0"
I20260929 02:06:07.915138 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "7633a7168c894b6396013d438f669589"
format_stamp: "Formatted at 2026-09-29 02:06:07 on dist-test-slave-q06p"
server_key: "a010635c2cdf5650bfca190a5a748174"
server_key_iv: "fc85674ace66b05df0f4e6af99a2dc95"
server_key_version: "encryptionkey@0"
I20260929 02:06:07.921962 14421 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 02:06:07.925937 14436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:07.927670 14421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:07.927877 14421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "7633a7168c894b6396013d438f669589"
format_stamp: "Formatted at 2026-09-29 02:06:07 on dist-test-slave-q06p"
server_key: "a010635c2cdf5650bfca190a5a748174"
server_key_iv: "fc85674ace66b05df0f4e6af99a2dc95"
server_key_version: "encryptionkey@0"
I20260929 02:06:07.928097 14421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:07.953243 14421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:07.957849 14421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:07.958155 14421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:07.976547 14421 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:41245
I20260929 02:06:07.976555 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:41245 every 8 connection(s)
I20260929 02:06:07.977999 14421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:07.979841 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 14421
I20260929 02:06:07.980106 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:07.980877 11995 external_mini_cluster.cc:1428] Setting key 8a3a497606f57c7a95e03320705eab5e
I20260929 02:06:07.983063 14488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:06:07 dist-test-slave-q06p krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:07.998705 14488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589: Bootstrap starting.
I20260929 02:06:08.003229 14488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:08.004638 14488 log.cc:824] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:08.008570 14488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589: No bootstrap required, opened a new log
I20260929 02:06:08.009770 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:07.983111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.11.182.254:41245)
0929 02:06:07.985049 (+ 1938us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:07.985062 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:07.985107 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:06:07.986682 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:07.986705 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:07.987252 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:07.987635 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:07.989590 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:07.991325 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:07.993238 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:07.994339 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:07.994517 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:07.999669 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:07.999681 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:07.999702 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:07.999759 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:08.004129 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:08.005480 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:08.005488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:08.005495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:08.005699 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:08.006238 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:08.006245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:08.006250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:08.007022 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:08.007334 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:08.007815 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 02:06:08.008291 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1688,"thread_start_us":1445,"threads_started":1}
I20260929 02:06:08.017488 14488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633a7168c894b6396013d438f669589" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41245 } }
I20260929 02:06:08.018006 14488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:08.018121 14488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7633a7168c894b6396013d438f669589, State: Initialized, Role: FOLLOWER
I20260929 02:06:08.018788 14488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633a7168c894b6396013d438f669589" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41245 } }
I20260929 02:06:08.019318 14488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:08.019469 14488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:08.019687 14488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:08.021946 14488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633a7168c894b6396013d438f669589" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41245 } }
I20260929 02:06:08.022527 14488 leader_election.cc:302] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7633a7168c894b6396013d438f669589; no voters:
I20260929 02:06:08.023260 14488 leader_election.cc:288] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:08.023425 14493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:08.024114 14493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [term 1 LEADER]: Becoming Leader. State: Replica: 7633a7168c894b6396013d438f669589, State: Running, Role: LEADER
I20260929 02:06:08.024744 14493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633a7168c894b6396013d438f669589" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41245 } }
I20260929 02:06:08.025673 14488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:08.028239 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7633a7168c894b6396013d438f669589. Latest consensus state: current_term: 1 leader_uuid: "7633a7168c894b6396013d438f669589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633a7168c894b6396013d438f669589" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41245 } } }
I20260929 02:06:08.028266 14494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7633a7168c894b6396013d438f669589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633a7168c894b6396013d438f669589" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41245 } } }
I20260929 02:06:08.028632 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:08.028698 14494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:08.030833 14498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:08.036605 14498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:08.046437 14498 catalog_manager.cc:1347] Generated new cluster ID: a77ac31712c04d91aab06b1c2e57517e
I20260929 02:06:08.046589 14498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:08.063822 14498 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:08.065864 14498 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:08.075955 14498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7633a7168c894b6396013d438f669589: Generated new TSK 0
I20260929 02:06:08.077077 14498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:06:08.143158 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41245
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:08.489482 14514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:08.489852 14514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:08.490021 14514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:08.497741 14514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:08.497920 14514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:08.498107 14514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:08.508363 14514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:08.509860 14514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:08.511303 14514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:08.524921 14519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:08.524932 14522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:08.525218 14520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:08.525840 14514 server_base.cc:1034] running on GCE node
I20260929 02:06:08.526472 14514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:08.527595 14514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:08.528815 14514 hybrid_clock.cc:648] HybridClock initialized: now 1790647568528714 us; error 69 us; skew 500 ppm
I20260929 02:06:08.529223 14514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:08 dist-test-slave-q06p krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647568, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:08.533303 14514 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:06:08.535005 14514 webserver.cc:460] Webserver started at http://127.11.182.193:46113/ using document root <none> and password file <none>
I20260929 02:06:08.535742 14514 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:08.535902 14514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:08.536221 14514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:08.539247 14514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "6ce514fbb11842b38ba5d0eac0772655"
format_stamp: "Formatted at 2026-09-29 02:06:08 on dist-test-slave-q06p"
server_key: "e633442bc52bcd3f3575cfcb7515d746"
server_key_iv: "e32685916273fb34b19c8429d8ed8b87"
server_key_version: "encryptionkey@0"
I20260929 02:06:08.540059 14514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "6ce514fbb11842b38ba5d0eac0772655"
format_stamp: "Formatted at 2026-09-29 02:06:08 on dist-test-slave-q06p"
server_key: "e633442bc52bcd3f3575cfcb7515d746"
server_key_iv: "e32685916273fb34b19c8429d8ed8b87"
server_key_version: "encryptionkey@0"
I20260929 02:06:08.545286 14514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:06:08.549172 14529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:08.550647 14514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:08.550838 14514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "6ce514fbb11842b38ba5d0eac0772655"
format_stamp: "Formatted at 2026-09-29 02:06:08 on dist-test-slave-q06p"
server_key: "e633442bc52bcd3f3575cfcb7515d746"
server_key_iv: "e32685916273fb34b19c8429d8ed8b87"
server_key_version: "encryptionkey@0"
I20260929 02:06:08.551164 14514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:08.566659 14514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:08.571395 14514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:08.571727 14514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:08.572765 14514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:08.574187 14514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:08.574303 14514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:08.574457 14514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:08.574533 14514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:08.608656 14514 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:40975
I20260929 02:06:08.608737 14641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:40975 every 8 connection(s)
I20260929 02:06:08.609966 14514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:08.613281 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 14514
I20260929 02:06:08.613504 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:08.614341 11995 external_mini_cluster.cc:1428] Setting key cc196e01ef01e7151f5fe5e15f3ffd6c
Sep 29 02:06:08 dist-test-slave-q06p krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647568, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:08.634023 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:08.612619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:37809 (local address 127.11.182.254:41245)
0929 02:06:08.613103 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:08.613111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:08.614448 (+ 1337us) server_negotiation.cc:407] Connection header received
0929 02:06:08.616652 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:08.616656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:08.616723 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:08.616813 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:08.619066 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:08.619764 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:08.622251 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:08.622980 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:08.623131 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:08.627303 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:08.627312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:08.627321 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:08.627355 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:08.630271 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:08.631464 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:08.631468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:08.631470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:08.631595 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:08.632177 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:08.632180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:08.632182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:08.632516 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:08.632624 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:08.633598 (+ 974us) server_negotiation.cc:299] Negotiation successful
0929 02:06:08.633767 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:06:08.635443 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:08.613218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41245 (local address 127.11.182.193:37809)
0929 02:06:08.614112 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:08.614175 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:08.616337 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:08.617113 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:08.617125 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:08.617959 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:08.618877 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:08.618909 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:08.619926 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:08.619934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:08.622107 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:08.622120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:08.623211 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:08.623216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:08.623428 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:08.624144 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:08.624166 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:08.626990 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:08.630470 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:08.630486 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:08.630511 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:08.631293 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:08.631758 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:08.631766 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:08.631771 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:08.632041 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:08.632657 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:08.632669 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:08.633158 (+ 489us) client_negotiation.cc:769] Sending connection context
0929 02:06:08.633508 (+ 350us) client_negotiation.cc:240] Negotiation successful
0929 02:06:08.634107 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":357,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
I20260929 02:06:08.637161 14642 heartbeater.cc:347] Connected to a master server at 127.11.182.254:41245
I20260929 02:06:08.637540 14642 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:08.638307 14642 heartbeater.cc:511] Master 127.11.182.254:41245 requested a full tablet report, sending...
I20260929 02:06:08.640763 14453 ts_manager.cc:194] Registered new tserver with Master: 6ce514fbb11842b38ba5d0eac0772655 (127.11.182.193:40975)
I20260929 02:06:08.642429 14453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:37809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:06:08.663241 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41245
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:09.011397 14648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:09.011698 14648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:09.011876 14648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:09.019972 14648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:09.020110 14648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:09.020233 14648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:09.031093 14648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:09.032658 14648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:09.034343 14648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:09.050730 14653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:09.050666 14656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:09.050717 14654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:09.052390 14648 server_base.cc:1034] running on GCE node
I20260929 02:06:09.053226 14648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:09.054430 14648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:09.055697 14648 hybrid_clock.cc:648] HybridClock initialized: now 1790647569055608 us; error 94 us; skew 500 ppm
I20260929 02:06:09.056151 14648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:09 dist-test-slave-q06p krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647569, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:09.059971 14648 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:06:09.062110 14648 webserver.cc:460] Webserver started at http://127.11.182.194:41043/ using document root <none> and password file <none>
I20260929 02:06:09.063398 14648 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:09.063580 14648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:09.063928 14648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:09.067312 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "72cc1226796a4d27a02233ea3c30048f"
format_stamp: "Formatted at 2026-09-29 02:06:09 on dist-test-slave-q06p"
server_key: "9af7f0070d51c6be7a47ec4e454066a3"
server_key_iv: "d63c6781f995daba0aba8f4ca3d02abf"
server_key_version: "encryptionkey@0"
I20260929 02:06:09.068238 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "72cc1226796a4d27a02233ea3c30048f"
format_stamp: "Formatted at 2026-09-29 02:06:09 on dist-test-slave-q06p"
server_key: "9af7f0070d51c6be7a47ec4e454066a3"
server_key_iv: "d63c6781f995daba0aba8f4ca3d02abf"
server_key_version: "encryptionkey@0"
I20260929 02:06:09.074419 14648 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 02:06:09.079216 14663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:09.080986 14648 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 02:06:09.081220 14648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "72cc1226796a4d27a02233ea3c30048f"
format_stamp: "Formatted at 2026-09-29 02:06:09 on dist-test-slave-q06p"
server_key: "9af7f0070d51c6be7a47ec4e454066a3"
server_key_iv: "d63c6781f995daba0aba8f4ca3d02abf"
server_key_version: "encryptionkey@0"
I20260929 02:06:09.081455 14648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:09.098335 14648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:09.103981 14648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:09.104317 14648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:09.105428 14648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:09.107041 14648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:09.107147 14648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:09.107298 14648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:09.107369 14648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:09.146538 14648 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:42657
I20260929 02:06:09.146587 14775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:42657 every 8 connection(s)
I20260929 02:06:09.147959 14648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:09.154019 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 14648
I20260929 02:06:09.154258 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:09.154691 11995 external_mini_cluster.cc:1428] Setting key b0ddda2d277bec94506dc6646f6a4c89
Sep 29 02:06:09 dist-test-slave-q06p krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647569, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:09.169546 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:09.150771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:59707 (local address 127.11.182.254:41245)
0929 02:06:09.151304 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:09.151308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:09.152336 (+ 1028us) server_negotiation.cc:407] Connection header received
0929 02:06:09.154330 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:09.154335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:09.154445 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:09.154570 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:09.156497 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:09.157239 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:09.158924 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:09.159350 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:09.159530 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:09.163298 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:09.163303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:09.163306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:09.163339 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:09.166288 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:09.167190 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:09.167194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:09.167197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:09.167325 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:09.167795 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:09.167799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:09.167802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:09.168058 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:09.168213 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:09.168931 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 02:06:09.169156 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":330,"threads_started":1}
I20260929 02:06:09.170595 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:09.151402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41245 (local address 127.11.182.194:59707)
0929 02:06:09.152161 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:09.152210 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:09.154019 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:09.154928 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:09.154941 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:09.155711 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:09.156321 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:09.156340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:09.157403 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:09.157409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:09.158724 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:09.158736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:09.159615 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:09.159624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:09.159906 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:09.160643 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:09.160666 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:09.163034 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:09.166475 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:09.166485 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:09.166506 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:09.167044 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:09.167455 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:09.167462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:09.167470 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:09.167659 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:09.168191 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:09.168199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:09.168562 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 02:06:09.168842 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 02:06:09.169409 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":268,"threads_started":1}
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:06:09.172407 14776 heartbeater.cc:347] Connected to a master server at 127.11.182.254:41245
I20260929 02:06:09.172742 14776 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:09.173511 14776 heartbeater.cc:511] Master 127.11.182.254:41245 requested a full tablet report, sending...
I20260929 02:06:09.175323 14453 ts_manager.cc:194] Registered new tserver with Master: 72cc1226796a4d27a02233ea3c30048f (127.11.182.194:42657)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
I20260929 02:06:09.176375 14453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:59707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:06:09.204162 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41245
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:09.550623 14782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:09.551056 14782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:09.551252 14782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:09.559904 14782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:09.560065 14782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:09.560268 14782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:09.570839 14782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:09.572247 14782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:09.573638 14782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:09.588928 14787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:09.588956 14788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:09.589035 14790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:09.589926 14782 server_base.cc:1034] running on GCE node
I20260929 02:06:09.590653 14782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:09.591817 14782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:09.593130 14782 hybrid_clock.cc:648] HybridClock initialized: now 1790647569593063 us; error 57 us; skew 500 ppm
I20260929 02:06:09.593549 14782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:09 dist-test-slave-q06p krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647569, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:09.597263 14782 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:06:09.601536 14782 webserver.cc:460] Webserver started at http://127.11.182.195:42871/ using document root <none> and password file <none>
I20260929 02:06:09.602653 14782 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:09.602864 14782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:09.603324 14782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:09.607925 14782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "98af16f3206e417c891b41405162ea44"
format_stamp: "Formatted at 2026-09-29 02:06:09 on dist-test-slave-q06p"
server_key: "81f843dbb6fef0bf8c9285262dfcc421"
server_key_iv: "24bc89589b743900da2f84cae7338c82"
server_key_version: "encryptionkey@0"
I20260929 02:06:09.608816 14782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "98af16f3206e417c891b41405162ea44"
format_stamp: "Formatted at 2026-09-29 02:06:09 on dist-test-slave-q06p"
server_key: "81f843dbb6fef0bf8c9285262dfcc421"
server_key_iv: "24bc89589b743900da2f84cae7338c82"
server_key_version: "encryptionkey@0"
I20260929 02:06:09.614400 14782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:06:09.618505 14797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:09.620131 14782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:09.620328 14782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "98af16f3206e417c891b41405162ea44"
format_stamp: "Formatted at 2026-09-29 02:06:09 on dist-test-slave-q06p"
server_key: "81f843dbb6fef0bf8c9285262dfcc421"
server_key_iv: "24bc89589b743900da2f84cae7338c82"
server_key_version: "encryptionkey@0"
I20260929 02:06:09.620622 14782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:09.645758 14782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:09.645975 14642 heartbeater.cc:503] Master 127.11.182.254:41245 was elected leader, sending a full tablet report...
I20260929 02:06:09.651815 14782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:09.652204 14782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:09.653652 14782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:09.656556 14782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:09.656656 14782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:09.656750 14782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:09.656848 14782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:09.697386 14782 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:42711
I20260929 02:06:09.697455 14909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:42711 every 8 connection(s)
I20260929 02:06:09.698827 14782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:09.705483 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 14782
I20260929 02:06:09.705704 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:09.706255 11995 external_mini_cluster.cc:1428] Setting key abd269f19cd4da95a6b8af0c07d6ee0b
Sep 29 02:06:09 dist-test-slave-q06p krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647569, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:09.723412 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:09.702338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:47009 (local address 127.11.182.254:41245)
0929 02:06:09.702839 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:09.702895 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:09.703898 (+ 1003us) server_negotiation.cc:407] Connection header received
0929 02:06:09.707152 (+ 3254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:09.707158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:09.707286 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:09.707414 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:09.709604 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:09.710517 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:09.712201 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:09.712807 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:09.712917 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:09.716649 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:09.716654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:09.716657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:09.716691 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:09.719601 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:09.720475 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:09.720481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:09.720484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:09.720635 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:09.721299 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:09.721305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:09.721307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:09.721621 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:09.721809 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:09.722636 (+ 827us) server_negotiation.cc:299] Negotiation successful
0929 02:06:09.722933 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":251,"threads_started":1}
I20260929 02:06:09.724632 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:09.702744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41245 (local address 127.11.182.195:47009)
0929 02:06:09.703619 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:09.703668 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:09.706718 (+ 3050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:09.707736 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:09.707750 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:09.708573 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:09.709394 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:09.709423 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:09.710663 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:09.710668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:09.712055 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:09.712068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:09.713027 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:09.713032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:09.713238 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:09.714115 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:09.714140 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:09.716380 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:09.719775 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:09.719786 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:09.719802 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:09.720309 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:09.720827 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:09.720833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:09.720837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:09.721136 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:09.721803 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:09.721812 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:09.722213 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 02:06:09.722519 (+ 306us) client_negotiation.cc:240] Negotiation successful
0929 02:06:09.723162 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":333,"threads_started":1}
I20260929 02:06:09.726272 14910 heartbeater.cc:347] Connected to a master server at 127.11.182.254:41245
I20260929 02:06:09.726613 14910 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:09.727439 14910 heartbeater.cc:511] Master 127.11.182.254:41245 requested a full tablet report, sending...
I20260929 02:06:09.728973 14453 ts_manager.cc:194] Registered new tserver with Master: 98af16f3206e417c891b41405162ea44 (127.11.182.195:42711)
I20260929 02:06:09.729779 14453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:47009
I20260929 02:06:09.735808 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:06:09.776119 14912 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:09.773358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.11.182.254:41245)
0929 02:06:09.773608 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:09.773617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:09.774067 (+ 450us) server_negotiation.cc:407] Connection header received
0929 02:06:09.774291 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:09.774296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:09.774362 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:09.774481 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:09.775867 (+ 1386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35814: BlockingRecv error: recv got EOF from 127.0.0.1:35814 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:06:09.779276 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14514
W20260929 02:06:09.782565 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14514: Not authorized: Client connection negotiation failed: client connection to 127.11.182.193:40975: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:06:09.782799 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 14514
I20260929 02:06:09.801635 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14648
W20260929 02:06:09.804467 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14648: Not authorized: Client connection negotiation failed: client connection to 127.11.182.194:42657: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:06:09.804661 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 14648
I20260929 02:06:09.820607 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14782
W20260929 02:06:09.823535 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14782: Not authorized: Client connection negotiation failed: client connection to 127.11.182.195:42711: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:06:09.823674 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 14782
I20260929 02:06:09.839378 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14421
I20260929 02:06:09.840579 14483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:09.941963 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 14421
2026-09-29T02:06:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3777 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:10 dist-test-slave-q06p krb5kdc[14928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:10 dist-test-slave-q06p krb5kdc[14928](info): set up 2 sockets
Sep 29 02:06:10 dist-test-slave-q06p krb5kdc[14928](info): commencing operation
krb5kdc: starting...
W20260929 02:06:11.051923 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:11 dist-test-slave-q06p krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:06:11.278985 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34945
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:34945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:11.620519 14941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:11.620826 14941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:11.621021 14941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:11.628579 14941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:11.628706 14941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:11.628785 14941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:11.628896 14941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:11.628962 14941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:11.639250 14941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:34945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:11.640647 14941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:11.641980 14941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:11.654321 14946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:11.654232 14949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:11.654337 14947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:11.655026 14941 server_base.cc:1034] running on GCE node
I20260929 02:06:11.655862 14941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:11.657431 14941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:11.658685 14941 hybrid_clock.cc:648] HybridClock initialized: now 1790647571658635 us; error 57 us; skew 500 ppm
I20260929 02:06:11.659124 14941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:11 dist-test-slave-q06p krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647571, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:11.668222 14941 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:06:11.669940 14941 webserver.cc:460] Webserver started at http://127.11.182.254:39005/ using document root <none> and password file <none>
I20260929 02:06:11.670680 14941 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:11.670833 14941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:11.671213 14941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:11.674230 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "619867bbba964344baa08e1f71bd7d3d"
format_stamp: "Formatted at 2026-09-29 02:06:11 on dist-test-slave-q06p"
server_key: "0ef2882f903d9107f2f0aa1de8a7d1c8"
server_key_iv: "796be279e4516d879efa0f7bca538dc7"
server_key_version: "encryptionkey@0"
I20260929 02:06:11.675153 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "619867bbba964344baa08e1f71bd7d3d"
format_stamp: "Formatted at 2026-09-29 02:06:11 on dist-test-slave-q06p"
server_key: "0ef2882f903d9107f2f0aa1de8a7d1c8"
server_key_iv: "796be279e4516d879efa0f7bca538dc7"
server_key_version: "encryptionkey@0"
I20260929 02:06:11.680467 14941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:06:11.683982 14956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:11.685395 14941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:06:11.685588 14941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "619867bbba964344baa08e1f71bd7d3d"
format_stamp: "Formatted at 2026-09-29 02:06:11 on dist-test-slave-q06p"
server_key: "0ef2882f903d9107f2f0aa1de8a7d1c8"
server_key_iv: "796be279e4516d879efa0f7bca538dc7"
server_key_version: "encryptionkey@0"
I20260929 02:06:11.685796 14941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:11.710304 14941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:11.714637 14941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:11.714974 14941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:11.734014 14941 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:34945
I20260929 02:06:11.734041 15007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:34945 every 8 connection(s)
I20260929 02:06:11.735528 14941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:11.739221 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 14941
I20260929 02:06:11.739477 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:11.740074 15008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:11.740288 11995 external_mini_cluster.cc:1428] Setting key 24d8a205ba17bb2dd8da8037c28dfbe2
I20260929 02:06:11.752842 15008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d: Bootstrap starting.
Sep 29 02:06:11 dist-test-slave-q06p krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:11.756592 15008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:11.757658 15008 log.cc:824] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:11.761114 15008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d: No bootstrap required, opened a new log
I20260929 02:06:11.762989 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:11.742303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.11.182.254:34945)
0929 02:06:11.743156 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:11.743167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:11.743217 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:06:11.745042 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:11.745065 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:11.745484 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:11.745924 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:11.747434 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:11.748688 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:11.749926 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:11.750650 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:11.750892 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:11.755278 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:11.755287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:11.755302 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:11.755343 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:11.758759 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:11.759701 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:11.759708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:11.759715 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:11.759921 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:11.760345 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:11.760349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:11.760351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:11.760866 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:11.761076 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:11.761538 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 02:06:11.761929 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":364,"threads_started":1}
I20260929 02:06:11.767371 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:06:11.770974 15008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619867bbba964344baa08e1f71bd7d3d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34945 } }
I20260929 02:06:11.771472 15008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:11.771588 15008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619867bbba964344baa08e1f71bd7d3d, State: Initialized, Role: FOLLOWER
I20260929 02:06:11.772282 15008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619867bbba964344baa08e1f71bd7d3d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34945 } }
I20260929 02:06:11.772581 15008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:11.772722 15008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:11.772912 15008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:11.776034 15008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619867bbba964344baa08e1f71bd7d3d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34945 } }
I20260929 02:06:11.776566 15008 leader_election.cc:302] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 619867bbba964344baa08e1f71bd7d3d; no voters:
I20260929 02:06:11.777262 15008 leader_election.cc:288] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:11.777376 15014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:11.778558 15014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [term 1 LEADER]: Becoming Leader. State: Replica: 619867bbba964344baa08e1f71bd7d3d, State: Running, Role: LEADER
I20260929 02:06:11.779181 15014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619867bbba964344baa08e1f71bd7d3d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34945 } }
I20260929 02:06:11.779836 15008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:11.782249 15016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 619867bbba964344baa08e1f71bd7d3d. Latest consensus state: current_term: 1 leader_uuid: "619867bbba964344baa08e1f71bd7d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619867bbba964344baa08e1f71bd7d3d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34945 } } }
I20260929 02:06:11.782694 15016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:11.783391 15015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "619867bbba964344baa08e1f71bd7d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619867bbba964344baa08e1f71bd7d3d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34945 } } }
I20260929 02:06:11.783735 15015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:11.791685 15019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:11.797596 15019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:11.808314 15019 catalog_manager.cc:1347] Generated new cluster ID: e0c0a16a60544142bfdca757d5321d9d
I20260929 02:06:11.808437 15019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:11.818584 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:11.810933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.11.182.254:34945)
0929 02:06:11.811086 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:11.811090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:11.811832 (+ 742us) server_negotiation.cc:407] Connection header received
0929 02:06:11.812046 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:11.812051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:11.812128 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:11.812245 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:11.813240 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:11.814006 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:11.815396 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:11.815814 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:11.815959 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:11.817084 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:11.817094 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:11.817097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:11.817746 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:11.817809 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:11.817831 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:11.817972 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:11.818108 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 02:06:11.818205 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 02:06:11.825265 15019 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:11.827085 15019 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:11.835737 15019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 619867bbba964344baa08e1f71bd7d3d: Generated new TSK 0
I20260929 02:06:11.836795 15019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:06:11.848363 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 14941
I20260929 02:06:11.849509 15003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:11.938550 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 14941
2026-09-29T02:06:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1990 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:12 dist-test-slave-q06p krb5kdc[15043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:12 dist-test-slave-q06p krb5kdc[15043](info): set up 2 sockets
Sep 29 02:06:12 dist-test-slave-q06p krb5kdc[15043](info): commencing operation
krb5kdc: starting...
W20260929 02:06:13.039734 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:13 dist-test-slave-q06p krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:06:13.248732 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:36285
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:13.577361 15056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:13.577643 15056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:13.577733 15056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:13.584817 15056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:13.584926 15056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:13.584972 15056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:13.585034 15056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:13.585079 15056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:13.594671 15056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:13.596084 15056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:13.597324 15056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:13.615767 15062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:13.615767 15061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:13.616209 15064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:13.617218 15056 server_base.cc:1034] running on GCE node
I20260929 02:06:13.617991 15056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:13.619558 15056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:13.620779 15056 hybrid_clock.cc:648] HybridClock initialized: now 1790647573620725 us; error 30 us; skew 500 ppm
I20260929 02:06:13.621140 15056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:13 dist-test-slave-q06p krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647573, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:13.624441 15056 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:06:13.626004 15056 webserver.cc:460] Webserver started at http://127.11.182.254:43525/ using document root <none> and password file <none>
I20260929 02:06:13.626704 15056 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:13.626922 15056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:13.627269 15056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:13.630326 15056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "7f1de6583bf5425e98bfedd8115475d3"
format_stamp: "Formatted at 2026-09-29 02:06:13 on dist-test-slave-q06p"
server_key: "61b0ce5158ff823c2239d7065e4857b8"
server_key_iv: "729cf2c2a55c7550fdd1a061632baace"
server_key_version: "encryptionkey@0"
I20260929 02:06:13.631165 15056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "7f1de6583bf5425e98bfedd8115475d3"
format_stamp: "Formatted at 2026-09-29 02:06:13 on dist-test-slave-q06p"
server_key: "61b0ce5158ff823c2239d7065e4857b8"
server_key_iv: "729cf2c2a55c7550fdd1a061632baace"
server_key_version: "encryptionkey@0"
I20260929 02:06:13.636411 15056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:06:13.639976 15071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:13.641357 15056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:06:13.641571 15056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "7f1de6583bf5425e98bfedd8115475d3"
format_stamp: "Formatted at 2026-09-29 02:06:13 on dist-test-slave-q06p"
server_key: "61b0ce5158ff823c2239d7065e4857b8"
server_key_iv: "729cf2c2a55c7550fdd1a061632baace"
server_key_version: "encryptionkey@0"
I20260929 02:06:13.641812 15056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:13.656044 15056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:13.660226 15056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:13.660530 15056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:13.679349 15056 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:36285
I20260929 02:06:13.679374 15122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:36285 every 8 connection(s)
I20260929 02:06:13.680809 15056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:13.684724 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:13.688575 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 15056
I20260929 02:06:13.688786 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:13.689576 11995 external_mini_cluster.cc:1428] Setting key 4b9ae47b72d5a8160813fd2c74627d92
I20260929 02:06:13.698213 15123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3: Bootstrap starting.
I20260929 02:06:13.702067 15123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:13.703454 15123 log.cc:824] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:06:13 dist-test-slave-q06p krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:13.706549 15123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3: No bootstrap required, opened a new log
I20260929 02:06:13.712638 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:13.691667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.11.182.254:36285)
0929 02:06:13.692455 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:13.692467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:13.692509 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:06:13.694040 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:13.694062 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:13.694471 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:13.694808 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:13.696509 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:13.697809 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:13.699629 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:13.700354 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:13.700484 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:13.705327 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:13.705335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:13.705350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:13.705399 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:13.708522 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:13.709380 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:13.709387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:13.709393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:13.709555 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:13.709942 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:13.709948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:13.709950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:13.710481 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:13.710714 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:13.711142 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 02:06:13.711581 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":365,"threads_started":1}
I20260929 02:06:13.715777 15123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1de6583bf5425e98bfedd8115475d3" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36285 } }
I20260929 02:06:13.716146 15123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:13.716252 15123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1de6583bf5425e98bfedd8115475d3, State: Initialized, Role: FOLLOWER
I20260929 02:06:13.716787 15123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1de6583bf5425e98bfedd8115475d3" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36285 } }
I20260929 02:06:13.717017 15123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:13.717157 15123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:13.717326 15123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:13.719750 15123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1de6583bf5425e98bfedd8115475d3" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36285 } }
I20260929 02:06:13.720237 15123 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1de6583bf5425e98bfedd8115475d3; no voters:
I20260929 02:06:13.720788 15123 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:13.720927 15128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:13.721632 15128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [term 1 LEADER]: Becoming Leader. State: Replica: 7f1de6583bf5425e98bfedd8115475d3, State: Running, Role: LEADER
I20260929 02:06:13.722137 15128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1de6583bf5425e98bfedd8115475d3" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36285 } }
I20260929 02:06:13.723001 15123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:13.725394 15130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1de6583bf5425e98bfedd8115475d3. Latest consensus state: current_term: 1 leader_uuid: "7f1de6583bf5425e98bfedd8115475d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1de6583bf5425e98bfedd8115475d3" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36285 } } }
I20260929 02:06:13.725826 15130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:13.729486 15135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:13.729384 15129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f1de6583bf5425e98bfedd8115475d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1de6583bf5425e98bfedd8115475d3" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36285 } } }
I20260929 02:06:13.729743 15129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:13.735056 15135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:13.744580 15135 catalog_manager.cc:1347] Generated new cluster ID: e9f18a1d7a134ab681a9df9edd1375cb
I20260929 02:06:13.744712 15135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:13.770092 15135 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:13.771636 15135 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:13.781513 15135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f1de6583bf5425e98bfedd8115475d3: Generated new TSK 0
I20260929 02:06:13.782334 15135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:06:13.838403 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36285
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:14.180409 15149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:14.180723 15149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:14.180819 15149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:14.188319 15149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:14.188443 15149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:14.188558 15149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:14.198670 15149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:14.200047 15149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:14.201324 15149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:14.215416 15155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:14.215498 15154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:14.215529 15157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:14.216315 15149 server_base.cc:1034] running on GCE node
I20260929 02:06:14.217008 15149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:14.218027 15149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:14.219275 15149 hybrid_clock.cc:648] HybridClock initialized: now 1790647574219206 us; error 71 us; skew 500 ppm
I20260929 02:06:14.219641 15149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:14 dist-test-slave-q06p krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647574, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:14.223230 15149 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:06:14.224942 15149 webserver.cc:460] Webserver started at http://127.11.182.193:37609/ using document root <none> and password file <none>
I20260929 02:06:14.225710 15149 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:14.225854 15149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:14.226176 15149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:14.229712 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "07b22c8db25c4ecc936a921e300ca6eb"
format_stamp: "Formatted at 2026-09-29 02:06:14 on dist-test-slave-q06p"
server_key: "13da8c34e2d8559f6285a5d372aa7d32"
server_key_iv: "3c870857b632870a583d26e88ebf88fa"
server_key_version: "encryptionkey@0"
I20260929 02:06:14.230648 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "07b22c8db25c4ecc936a921e300ca6eb"
format_stamp: "Formatted at 2026-09-29 02:06:14 on dist-test-slave-q06p"
server_key: "13da8c34e2d8559f6285a5d372aa7d32"
server_key_iv: "3c870857b632870a583d26e88ebf88fa"
server_key_version: "encryptionkey@0"
I20260929 02:06:14.236397 15149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:06:14.240223 15164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:14.241799 15149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:06:14.242013 15149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "07b22c8db25c4ecc936a921e300ca6eb"
format_stamp: "Formatted at 2026-09-29 02:06:14 on dist-test-slave-q06p"
server_key: "13da8c34e2d8559f6285a5d372aa7d32"
server_key_iv: "3c870857b632870a583d26e88ebf88fa"
server_key_version: "encryptionkey@0"
I20260929 02:06:14.242236 15149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:14.261456 15149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:14.266151 15149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:14.266489 15149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:14.267606 15149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:14.269104 15149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:14.269186 15149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:14.269338 15149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:14.269400 15149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:14.303344 15149 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:35449
I20260929 02:06:14.303395 15276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:35449 every 8 connection(s)
I20260929 02:06:14.304633 15149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:14.308591 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 15149
I20260929 02:06:14.308802 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:14.309429 11995 external_mini_cluster.cc:1428] Setting key 39f0a61ec8f27fb548af8ff958805718
Sep 29 02:06:14 dist-test-slave-q06p krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647574, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:14.325861 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:14.307233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:36933 (local address 127.11.182.254:36285)
0929 02:06:14.307688 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:14.307692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:14.308568 (+ 876us) server_negotiation.cc:407] Connection header received
0929 02:06:14.310681 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:14.310687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:14.310770 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:14.310912 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:14.312603 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:14.313312 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:14.314993 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:14.315410 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:14.315552 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:14.319571 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:14.319576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:14.319580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:14.319615 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:14.322316 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:14.323312 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:14.323320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:14.323325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:14.323458 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:14.323929 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:14.323934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:14.323939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:14.324239 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:14.324450 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:14.325240 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 02:06:14.325473 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":253,"threads_started":1}
I20260929 02:06:14.326941 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:14.307678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36285 (local address 127.11.182.193:36933)
0929 02:06:14.308399 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:14.308438 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:14.310409 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:14.311241 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:14.311254 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:14.311821 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:14.312414 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:14.312432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:14.313509 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:14.313514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:14.314777 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:14.314790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:14.315658 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:14.315665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:14.315917 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:14.316667 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:14.316695 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:14.319285 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:14.322524 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:14.322534 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:14.322565 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:14.323125 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:14.323586 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:14.323591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:14.323594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:14.323794 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:14.324422 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:14.324430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:14.324850 (+ 420us) client_negotiation.cc:769] Sending connection context
0929 02:06:14.325127 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 02:06:14.325717 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":289,"threads_started":1}
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:06:14.328589 15277 heartbeater.cc:347] Connected to a master server at 127.11.182.254:36285
I20260929 02:06:14.329044 15277 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:14.330001 15277 heartbeater.cc:511] Master 127.11.182.254:36285 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
I20260929 02:06:14.332726 15088 ts_manager.cc:194] Registered new tserver with Master: 07b22c8db25c4ecc936a921e300ca6eb (127.11.182.193:35449)
I20260929 02:06:14.334981 15088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:36933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:06:14.359714 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36285
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:14.703684 15283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:14.703997 15283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:14.704154 15283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:14.711709 15283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:14.711827 15283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:14.712021 15283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:14.721948 15283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:14.723471 15283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:14.724920 15283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:14.738521 15288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:14.738433 15291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:14.738386 15289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:14.740190 15283 server_base.cc:1034] running on GCE node
I20260929 02:06:14.740765 15283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:14.741686 15283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:14.742973 15283 hybrid_clock.cc:648] HybridClock initialized: now 1790647574742915 us; error 36 us; skew 500 ppm
I20260929 02:06:14.743330 15283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:14 dist-test-slave-q06p krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647574, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:14.746977 15283 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:06:14.748492 15283 webserver.cc:460] Webserver started at http://127.11.182.194:33469/ using document root <none> and password file <none>
I20260929 02:06:14.749159 15283 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:14.749264 15283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:14.749521 15283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:14.752434 15283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "d9941a170c4a4634acfe851000944cdc"
format_stamp: "Formatted at 2026-09-29 02:06:14 on dist-test-slave-q06p"
server_key: "62cd04461aad0b049db7d1cd2f1e5beb"
server_key_iv: "1cfd51ddb19db4312c18c9aee8ac0910"
server_key_version: "encryptionkey@0"
I20260929 02:06:14.753170 15283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "d9941a170c4a4634acfe851000944cdc"
format_stamp: "Formatted at 2026-09-29 02:06:14 on dist-test-slave-q06p"
server_key: "62cd04461aad0b049db7d1cd2f1e5beb"
server_key_iv: "1cfd51ddb19db4312c18c9aee8ac0910"
server_key_version: "encryptionkey@0"
I20260929 02:06:14.758399 15283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:06:14.762214 15298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:14.763801 15283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:14.764024 15283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "d9941a170c4a4634acfe851000944cdc"
format_stamp: "Formatted at 2026-09-29 02:06:14 on dist-test-slave-q06p"
server_key: "62cd04461aad0b049db7d1cd2f1e5beb"
server_key_iv: "1cfd51ddb19db4312c18c9aee8ac0910"
server_key_version: "encryptionkey@0"
I20260929 02:06:14.764228 15283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:14.786686 15283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:14.791316 15283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:14.791630 15283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:14.792629 15283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:14.793987 15283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:14.794061 15283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:14.794224 15283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:14.794284 15283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:14.829267 15283 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:44253
I20260929 02:06:14.829358 15410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:44253 every 8 connection(s)
I20260929 02:06:14.830632 15283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:14.840332 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 15283
I20260929 02:06:14.840525 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:14.841049 11995 external_mini_cluster.cc:1428] Setting key 48e72e6c3087212eb79dfbe7053471c1
Sep 29 02:06:14 dist-test-slave-q06p krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647574, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:14.853179 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:14.833251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:42473 (local address 127.11.182.254:36285)
0929 02:06:14.833669 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:14.833675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:14.834739 (+ 1064us) server_negotiation.cc:407] Connection header received
0929 02:06:14.836625 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:14.836629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:14.836696 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:14.836793 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:14.838302 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:14.839130 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:14.840886 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:14.841480 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:14.841625 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:14.846284 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:14.846288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:14.846291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:14.846323 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:14.849227 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:14.850338 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:14.850342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:14.850344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:14.850481 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:14.851101 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:14.851108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:14.851114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:14.851507 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:14.851660 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:14.852422 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 02:06:14.852686 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":192,"threads_started":1}
I20260929 02:06:14.853849 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:14.833887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36285 (local address 127.11.182.194:42473)
0929 02:06:14.834571 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:14.834616 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:14.836311 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:14.837033 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:14.837041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:14.837551 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:14.838116 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:14.838134 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:14.839302 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:14.839306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:14.840733 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:14.840753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:14.841796 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:14.841803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:14.842083 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:14.842917 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:14.842949 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:14.846043 (+ 3094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:14.849415 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:14.849430 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:14.849455 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:14.850196 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:14.850609 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:14.850617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:14.850622 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:14.850947 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:14.851657 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:14.851664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:14.852022 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 02:06:14.852261 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 02:06:14.852680 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":257,"threads_started":1}
I20260929 02:06:14.855369 15411 heartbeater.cc:347] Connected to a master server at 127.11.182.254:36285
I20260929 02:06:14.855695 15411 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:14.856437 15411 heartbeater.cc:511] Master 127.11.182.254:36285 requested a full tablet report, sending...
I20260929 02:06:14.857961 15088 ts_manager.cc:194] Registered new tserver with Master: d9941a170c4a4634acfe851000944cdc (127.11.182.194:44253)
I20260929 02:06:14.858773 15088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:42473
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:06:14.889916 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36285
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:15.231179 15417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:15.231485 15417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:15.231655 15417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:15.240037 15417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:15.240168 15417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:15.240366 15417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:15.251849 15417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:15.253264 15417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:15.254729 15417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:15.269673 15422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:15.269721 15423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:15.270171 15425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:15.270638 15417 server_base.cc:1034] running on GCE node
I20260929 02:06:15.271356 15417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:15.272357 15417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:15.273576 15417 hybrid_clock.cc:648] HybridClock initialized: now 1790647575273513 us; error 56 us; skew 500 ppm
I20260929 02:06:15.273975 15417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647575, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:15.277263 15417 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:06:15.279095 15417 webserver.cc:460] Webserver started at http://127.11.182.195:44683/ using document root <none> and password file <none>
I20260929 02:06:15.279872 15417 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:15.280011 15417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:15.280349 15417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:15.283553 15417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "e232fef816094366a5e81fce390e2e81"
format_stamp: "Formatted at 2026-09-29 02:06:15 on dist-test-slave-q06p"
server_key: "713d78ca6eab653bc12cf5708e8eb616"
server_key_iv: "52dff349c87bfcff2631a44ee850bbb9"
server_key_version: "encryptionkey@0"
I20260929 02:06:15.284407 15417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "e232fef816094366a5e81fce390e2e81"
format_stamp: "Formatted at 2026-09-29 02:06:15 on dist-test-slave-q06p"
server_key: "713d78ca6eab653bc12cf5708e8eb616"
server_key_iv: "52dff349c87bfcff2631a44ee850bbb9"
server_key_version: "encryptionkey@0"
I20260929 02:06:15.290045 15417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:06:15.294067 15432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:15.295811 15417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:15.296103 15417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "e232fef816094366a5e81fce390e2e81"
format_stamp: "Formatted at 2026-09-29 02:06:15 on dist-test-slave-q06p"
server_key: "713d78ca6eab653bc12cf5708e8eb616"
server_key_iv: "52dff349c87bfcff2631a44ee850bbb9"
server_key_version: "encryptionkey@0"
I20260929 02:06:15.296324 15417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:15.336148 15417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:15.339157 15277 heartbeater.cc:503] Master 127.11.182.254:36285 was elected leader, sending a full tablet report...
I20260929 02:06:15.341254 15417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:15.341563 15417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:15.342549 15417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:15.344022 15417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:15.344137 15417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:15.344295 15417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:15.344367 15417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:15.378778 15417 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:36563
I20260929 02:06:15.378829 15544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:36563 every 8 connection(s)
I20260929 02:06:15.380182 15417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:15.380736 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 15417
I20260929 02:06:15.380971 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:15.381611 11995 external_mini_cluster.cc:1428] Setting key 5b1752e044814f11eb06df5aa4a49c3c
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647575, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:15.402158 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:15.383411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:40449 (local address 127.11.182.254:36285)
0929 02:06:15.383939 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:15.383945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:15.385197 (+ 1252us) server_negotiation.cc:407] Connection header received
0929 02:06:15.387227 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:15.387231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:15.387375 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:15.387479 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:15.389056 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.389883 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.391609 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.392004 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.392101 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:15.395650 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:15.395655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:15.395658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:15.395695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:15.398784 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.399643 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.399647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.399650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.399840 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.400469 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.400473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.400475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.400814 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:15.400986 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:15.401715 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 02:06:15.401891 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":271,"threads_started":1}
I20260929 02:06:15.403263 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:15.384189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36285 (local address 127.11.182.195:40449)
0929 02:06:15.385008 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:15.385060 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:15.386924 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:15.387700 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:15.387709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:15.388261 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:15.388862 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:15.388880 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.390087 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.390091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:15.391480 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:15.391493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.392240 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.392245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:15.392496 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:15.393154 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:15.393177 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:15.395405 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:15.398981 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:15.398989 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:15.399005 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:15.399489 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:15.399992 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:15.399996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:15.399998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:15.400281 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:15.400948 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:15.400955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:15.401356 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 02:06:15.401602 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 02:06:15.402029 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":306,"threads_started":1}
I20260929 02:06:15.404928 15545 heartbeater.cc:347] Connected to a master server at 127.11.182.254:36285
I20260929 02:06:15.405244 15545 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:15.406009 15545 heartbeater.cc:511] Master 127.11.182.254:36285 requested a full tablet report, sending...
I20260929 02:06:15.407541 15088 ts_manager.cc:194] Registered new tserver with Master: e232fef816094366a5e81fce390e2e81 (127.11.182.195:36563)
I20260929 02:06:15.408304 15088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:40449
I20260929 02:06:15.410835 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647575, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647575, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:15.467291 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:15.451148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.11.182.254:36285)
0929 02:06:15.451496 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:15.451512 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:15.452128 (+ 616us) server_negotiation.cc:407] Connection header received
0929 02:06:15.452206 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:15.452209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:15.452283 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:15.452400 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:15.453867 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.454697 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.456211 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.456666 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.456805 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:15.461067 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:15.461073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:15.461076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:15.461119 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:15.464331 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.465210 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.465215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.465217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.465374 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.465892 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.465895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.465897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.466227 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:15.466416 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:15.466729 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 02:06:15.467007 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 02:06:15.467905 15088 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36022
I20260929 02:06:15.469432 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15149
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647575, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:06:15.486441 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:15.470498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.11.182.193:35449)
0929 02:06:15.471108 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:15.471119 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:15.471150 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:06:15.471221 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:15.471227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:15.471443 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:15.471610 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:15.472861 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.473876 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.475166 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.476146 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.476298 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:15.480540 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:15.480556 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:15.480572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:15.480616 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:15.483574 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.484317 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.484324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.484327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.484453 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.484852 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.484864 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.484866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.485207 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:15.485516 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:15.485814 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 02:06:15.486082 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":262,"threads_started":1}
W20260929 02:06:15.487927 15272 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39330
W20260929 02:06:15.489106 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15149: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:06:15.489338 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 15149
I20260929 02:06:15.505333 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15283
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647575, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:06:15.522178 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:15.506492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.11.182.194:44253)
0929 02:06:15.507302 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:15.507311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:15.507339 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:06:15.507410 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:15.507416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:15.507651 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:15.507794 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:15.509089 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.510112 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.511428 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.512011 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.512115 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:15.516381 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:15.516394 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:15.516409 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:15.516458 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:15.519549 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.520244 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.520253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.520255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.520380 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.520686 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.520689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.520691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.521044 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:15.521262 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:15.521556 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 02:06:15.521778 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":568,"threads_started":1}
W20260929 02:06:15.523675 15406 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40542
W20260929 02:06:15.524930 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15283: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:06:15.525168 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 15283
I20260929 02:06:15.540854 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15417
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647575, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:06:15.558924 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:15.541832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.11.182.195:36563)
0929 02:06:15.542660 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:15.542668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:15.542705 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:06:15.542767 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:15.542774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:15.543118 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:15.543316 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:15.544776 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.545986 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.547595 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:15.548231 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:15.548390 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:15.553000 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:15.553013 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:15.553029 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:15.553071 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:15.555978 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.556726 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.556733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.556736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.556861 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:15.557242 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:15.557253 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:15.557255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:15.557597 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:15.557918 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:15.558223 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 02:06:15.558498 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":492,"threads_started":1}
W20260929 02:06:15.560338 15540 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46766
W20260929 02:06:15.561502 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15417: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:06:15.561710 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 15417
I20260929 02:06:15.576851 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15056
I20260929 02:06:15.577998 15118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:15.682439 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 15056
2026-09-29T02:06:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3746 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15563](info): set up 2 sockets
Sep 29 02:06:15 dist-test-slave-q06p krb5kdc[15563](info): commencing operation
krb5kdc: starting...
W20260929 02:06:16.776865 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:16 dist-test-slave-q06p krb5kdc[15563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:06:17.000485 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40823
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:40823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:17.333873 15576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:17.334268 15576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:17.334448 15576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:17.341933 15576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:17.342105 15576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:17.342223 15576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:17.342320 15576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:17.342408 15576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:17.352794 15576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:40823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:17.354224 15576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:17.355635 15576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:17.368860 15584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:17.368970 15582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:17.369133 15581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:17.369763 15576 server_base.cc:1034] running on GCE node
I20260929 02:06:17.370695 15576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:17.372285 15576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:17.373593 15576 hybrid_clock.cc:648] HybridClock initialized: now 1790647577373532 us; error 55 us; skew 500 ppm
I20260929 02:06:17.374045 15576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:17 dist-test-slave-q06p krb5kdc[15563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647577, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:17.382941 15576 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:06:17.384792 15576 webserver.cc:460] Webserver started at http://127.11.182.254:36541/ using document root <none> and password file <none>
I20260929 02:06:17.385599 15576 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:17.385763 15576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:17.386086 15576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:17.389385 15576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "aa170a36219b4536a709d055c3561b57"
format_stamp: "Formatted at 2026-09-29 02:06:17 on dist-test-slave-q06p"
server_key: "f73a26437556d5685e1c61f82c262f16"
server_key_iv: "d0c2d5fcf4cb6b2a171d23febdc9c2b5"
server_key_version: "encryptionkey@0"
I20260929 02:06:17.390297 15576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "aa170a36219b4536a709d055c3561b57"
format_stamp: "Formatted at 2026-09-29 02:06:17 on dist-test-slave-q06p"
server_key: "f73a26437556d5685e1c61f82c262f16"
server_key_iv: "d0c2d5fcf4cb6b2a171d23febdc9c2b5"
server_key_version: "encryptionkey@0"
I20260929 02:06:17.396291 15576 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260929 02:06:17.400226 15591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:17.401782 15576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:06:17.402007 15576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "aa170a36219b4536a709d055c3561b57"
format_stamp: "Formatted at 2026-09-29 02:06:17 on dist-test-slave-q06p"
server_key: "f73a26437556d5685e1c61f82c262f16"
server_key_iv: "d0c2d5fcf4cb6b2a171d23febdc9c2b5"
server_key_version: "encryptionkey@0"
I20260929 02:06:17.402236 15576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:17.421057 15576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:17.425872 15576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:17.426192 15576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:17.447604 15576 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:40823
I20260929 02:06:17.447634 15642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:40823 every 8 connection(s)
I20260929 02:06:17.449083 15576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:17.451527 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 15576
I20260929 02:06:17.451814 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:17.452400 11995 external_mini_cluster.cc:1428] Setting key dd100c695f7cff4274364bd2060c053c
I20260929 02:06:17.453433 15643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:17.467813 15643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57: Bootstrap starting.
Sep 29 02:06:17 dist-test-slave-q06p krb5kdc[15563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:17.472041 15643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:17.473464 15643 log.cc:824] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:17.477214 15643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57: No bootstrap required, opened a new log
I20260929 02:06:17.478256 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:17.454428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.11.182.254:40823)
0929 02:06:17.455258 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:17.455275 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:17.455328 (+ 53us) server_negotiation.cc:407] Connection header received
0929 02:06:17.457150 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:17.457182 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:17.457769 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:17.458253 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:17.459752 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:17.461868 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:17.463315 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:17.464409 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:17.464609 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:17.468928 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:17.468939 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:17.468958 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:17.469011 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:17.473214 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:17.474150 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:17.474162 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:17.474172 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:17.474347 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:17.474806 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:17.474809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:17.474811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:17.475596 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:17.475928 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:17.476392 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 02:06:17.476836 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":357,"threads_started":1}
I20260929 02:06:17.485898 15643 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa170a36219b4536a709d055c3561b57" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40823 } }
I20260929 02:06:17.486268 15643 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:17.486335 15643 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa170a36219b4536a709d055c3561b57, State: Initialized, Role: FOLLOWER
I20260929 02:06:17.486836 15643 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa170a36219b4536a709d055c3561b57" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40823 } }
I20260929 02:06:17.487103 15643 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:17.487208 15643 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:17.487339 15643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:17.489408 15643 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa170a36219b4536a709d055c3561b57" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40823 } }
I20260929 02:06:17.489863 15643 leader_election.cc:302] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa170a36219b4536a709d055c3561b57; no voters:
I20260929 02:06:17.490433 15643 leader_election.cc:288] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:17.490667 15648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:17.491933 15648 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [term 1 LEADER]: Becoming Leader. State: Replica: aa170a36219b4536a709d055c3561b57, State: Running, Role: LEADER
I20260929 02:06:17.492445 15648 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa170a36219b4536a709d055c3561b57" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40823 } }
I20260929 02:06:17.492803 15643 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:17.495373 15650 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa170a36219b4536a709d055c3561b57. Latest consensus state: current_term: 1 leader_uuid: "aa170a36219b4536a709d055c3561b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa170a36219b4536a709d055c3561b57" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40823 } } }
I20260929 02:06:17.495805 15650 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:17.496342 15649 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa170a36219b4536a709d055c3561b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa170a36219b4536a709d055c3561b57" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40823 } } }
I20260929 02:06:17.496699 15649 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:17.496858 15657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:17.502641 15657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:17.511868 15657 catalog_manager.cc:1347] Generated new cluster ID: 3e3fbb5ee7d0435ea77a82a2c996eb91
I20260929 02:06:17.512001 15657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:17.521257 15657 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:17.522631 15657 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:17.535240 15657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa170a36219b4536a709d055c3561b57: Generated new TSK 0
I20260929 02:06:17.536440 15657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:06:17.595583 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40823
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:17.949405 15669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:17.949704 15669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:17.949800 15669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:17.957258 15669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:17.957387 15669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:17.957500 15669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:17.967499 15669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:17.968911 15669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:17.970181 15669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:17.984313 15675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:17.984411 15677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:17.984347 15674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:17.985065 15669 server_base.cc:1034] running on GCE node
I20260929 02:06:17.985795 15669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:17.986904 15669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:17.988134 15669 hybrid_clock.cc:648] HybridClock initialized: now 1790647577988092 us; error 35 us; skew 500 ppm
I20260929 02:06:17.988605 15669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:17 dist-test-slave-q06p krb5kdc[15563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647577, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:17.992023 15669 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:06:17.993659 15669 webserver.cc:460] Webserver started at http://127.11.182.193:37441/ using document root <none> and password file <none>
I20260929 02:06:17.994486 15669 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:17.994632 15669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:17.994987 15669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:17.998225 15669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "42e68acb2f0c47d2a0addf1f37df34e6"
format_stamp: "Formatted at 2026-09-29 02:06:17 on dist-test-slave-q06p"
server_key: "51a85ba965cd955f4fc717aa587af06b"
server_key_iv: "f9e675ca55135332d5a54601dd3f6943"
server_key_version: "encryptionkey@0"
I20260929 02:06:17.999168 15669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "42e68acb2f0c47d2a0addf1f37df34e6"
format_stamp: "Formatted at 2026-09-29 02:06:17 on dist-test-slave-q06p"
server_key: "51a85ba965cd955f4fc717aa587af06b"
server_key_iv: "f9e675ca55135332d5a54601dd3f6943"
server_key_version: "encryptionkey@0"
I20260929 02:06:18.005127 15669 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 02:06:18.008944 15684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:18.010658 15669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:18.010950 15669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "42e68acb2f0c47d2a0addf1f37df34e6"
format_stamp: "Formatted at 2026-09-29 02:06:17 on dist-test-slave-q06p"
server_key: "51a85ba965cd955f4fc717aa587af06b"
server_key_iv: "f9e675ca55135332d5a54601dd3f6943"
server_key_version: "encryptionkey@0"
I20260929 02:06:18.011191 15669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:18.037638 15669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:18.043483 15669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:18.043833 15669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:18.044962 15669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:18.046618 15669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:18.046741 15669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:18.046931 15669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:18.047013 15669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:18.082442 15669 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:46819
I20260929 02:06:18.082502 15796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:46819 every 8 connection(s)
I20260929 02:06:18.083879 15669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:18.090750 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 15669
I20260929 02:06:18.091024 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:18.091651 11995 external_mini_cluster.cc:1428] Setting key 7b8271834fe7bf7565ed3d807250da41
Sep 29 02:06:18 dist-test-slave-q06p krb5kdc[15563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647577, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:18.108350 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:18.086579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:35399 (local address 127.11.182.254:40823)
0929 02:06:18.087094 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:18.087098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:18.088100 (+ 1002us) server_negotiation.cc:407] Connection header received
0929 02:06:18.089970 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:18.089974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:18.090045 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:18.090150 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:18.092103 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:18.093134 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:18.095195 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:18.096024 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:18.096182 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:18.100360 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:18.100371 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:18.100379 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:18.100436 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:18.104231 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:18.105463 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:18.105468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:18.105470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:18.105620 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:18.106255 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:18.106261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:18.106266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:18.106618 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:18.106780 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:18.107685 (+ 905us) server_negotiation.cc:299] Negotiation successful
0929 02:06:18.107939 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":288,"threads_started":1}
I20260929 02:06:18.109416 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:18.087186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40823 (local address 127.11.182.193:35399)
0929 02:06:18.087934 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:18.087974 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:18.089684 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:18.090418 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:18.090427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:18.091169 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:18.091885 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:18.091909 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:18.093340 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:18.093345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:18.095029 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:18.095043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:18.096339 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:18.096347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:18.096704 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:18.097535 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:18.097560 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:18.099957 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:18.104466 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:18.104481 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:18.104506 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:18.105305 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:18.105841 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:18.105847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:18.105851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:18.106120 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:18.106763 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:18.106773 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:18.107283 (+ 510us) client_negotiation.cc:769] Sending connection context
0929 02:06:18.107583 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 02:06:18.108215 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":288,"threads_started":1}
I20260929 02:06:18.111332 15797 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40823
I20260929 02:06:18.111675 15797 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:06:18.112490 15797 heartbeater.cc:511] Master 127.11.182.254:40823 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
I20260929 02:06:18.115263 15608 ts_manager.cc:194] Registered new tserver with Master: 42e68acb2f0c47d2a0addf1f37df34e6 (127.11.182.193:46819)
I20260929 02:06:18.117010 15608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:35399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:06:18.143781 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40823
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:18.488593 15803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:18.488919 15803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:18.489089 15803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:18.497002 15803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:18.497130 15803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:18.497316 15803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:18.507746 15803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:18.509166 15803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:18.510504 15803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:18.525017 15808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:18.524994 15809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:18.525213 15811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:18.525791 15803 server_base.cc:1034] running on GCE node
I20260929 02:06:18.526531 15803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:18.527696 15803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:18.528954 15803 hybrid_clock.cc:648] HybridClock initialized: now 1790647578528903 us; error 43 us; skew 500 ppm
I20260929 02:06:18.529317 15803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:18 dist-test-slave-q06p krb5kdc[15563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647578, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:18.532621 15803 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:06:18.534323 15803 webserver.cc:460] Webserver started at http://127.11.182.194:39425/ using document root <none> and password file <none>
I20260929 02:06:18.535115 15803 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:18.535220 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:18.535593 15803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:18.538655 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "af340120b0d847cf93d2c926c90f6363"
format_stamp: "Formatted at 2026-09-29 02:06:18 on dist-test-slave-q06p"
server_key: "0315ef3f8dc2633434ce3375e0733a6c"
server_key_iv: "60ab43c52bbe8b942e0590f31977b619"
server_key_version: "encryptionkey@0"
I20260929 02:06:18.539568 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "af340120b0d847cf93d2c926c90f6363"
format_stamp: "Formatted at 2026-09-29 02:06:18 on dist-test-slave-q06p"
server_key: "0315ef3f8dc2633434ce3375e0733a6c"
server_key_iv: "60ab43c52bbe8b942e0590f31977b619"
server_key_version: "encryptionkey@0"
I20260929 02:06:18.545231 15803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:06:18.549492 15818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:18.551295 15803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:18.551512 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "af340120b0d847cf93d2c926c90f6363"
format_stamp: "Formatted at 2026-09-29 02:06:18 on dist-test-slave-q06p"
server_key: "0315ef3f8dc2633434ce3375e0733a6c"
server_key_iv: "60ab43c52bbe8b942e0590f31977b619"
server_key_version: "encryptionkey@0"
I20260929 02:06:18.551771 15803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:18.590533 15803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:18.595711 15803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:18.596076 15803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:18.597222 15803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:18.599994 15803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:18.600119 15803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:18.600340 15803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:18.600509 15803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:18.644068 15803 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:36685
I20260929 02:06:18.644126 15930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:36685 every 8 connection(s)
I20260929 02:06:18.645684 15803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:18.656118 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 15803
I20260929 02:06:18.656347 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:18.656865 11995 external_mini_cluster.cc:1428] Setting key 293fc515a7e8491e1ee4195fca591046
Sep 29 02:06:18 dist-test-slave-q06p krb5kdc[15563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647578, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:18.668728 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:18.648653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:51295 (local address 127.11.182.254:40823)
0929 02:06:18.649149 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:18.649153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:18.650215 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 02:06:18.652207 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:18.652213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:18.652318 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:18.652437 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:18.654228 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:18.655329 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:18.657788 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:18.658288 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:18.658432 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:18.662748 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:18.662754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:18.662757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:18.662795 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:18.665444 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:18.666295 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:18.666302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:18.666305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:18.666444 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:18.666913 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:18.666918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:18.666922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:18.667222 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:18.667386 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:18.668133 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 02:06:18.668356 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":291,"threads_started":1}
I20260929 02:06:18.669807 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:18.649214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40823 (local address 127.11.182.194:51295)
0929 02:06:18.650016 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:18.650077 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:18.651898 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:18.652705 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:18.652715 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:18.653287 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:18.653955 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:18.654002 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:18.655491 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:18.655496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:18.657626 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:18.657641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:18.658541 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:18.658549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:18.658804 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:18.659608 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:18.659640 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:18.662473 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:18.665615 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:18.665626 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:18.665652 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:18.666139 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:18.666567 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:18.666573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:18.666577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:18.666771 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:18.667367 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:18.667375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:18.667753 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 02:06:18.668034 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 02:06:18.668630 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":317,"threads_started":1}
I20260929 02:06:18.671347 15931 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40823
I20260929 02:06:18.671674 15931 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:18.672448 15931 heartbeater.cc:511] Master 127.11.182.254:40823 requested a full tablet report, sending...
I20260929 02:06:18.674072 15608 ts_manager.cc:194] Registered new tserver with Master: af340120b0d847cf93d2c926c90f6363 (127.11.182.194:36685)
I20260929 02:06:18.675537 15608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:51295
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:06:18.708036 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40823
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:19.058297 15937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:19.058604 15937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:19.058739 15937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:19.066996 15937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:19.067183 15937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:19.067396 15937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:19.078014 15937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:19.079418 15937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:19.080799 15937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:19.095542 15942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:19.095536 15945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:19.095607 15943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:19.096798 15937 server_base.cc:1034] running on GCE node
I20260929 02:06:19.097466 15937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:19.098580 15937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:19.099893 15937 hybrid_clock.cc:648] HybridClock initialized: now 1790647579099817 us; error 81 us; skew 500 ppm
I20260929 02:06:19.100288 15937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[15563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647579, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:19.103847 15937 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:06:19.105643 15937 webserver.cc:460] Webserver started at http://127.11.182.195:35795/ using document root <none> and password file <none>
I20260929 02:06:19.106422 15937 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:19.106575 15937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:19.106980 15937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:19.110095 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "8c93f83496bc49cbb06285f187053ba7"
format_stamp: "Formatted at 2026-09-29 02:06:19 on dist-test-slave-q06p"
server_key: "27c394b3d5f8356912284e3fba95b901"
server_key_iv: "1a42e0567c9d4c09d9f9597c1274dfa3"
server_key_version: "encryptionkey@0"
I20260929 02:06:19.111052 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "8c93f83496bc49cbb06285f187053ba7"
format_stamp: "Formatted at 2026-09-29 02:06:19 on dist-test-slave-q06p"
server_key: "27c394b3d5f8356912284e3fba95b901"
server_key_iv: "1a42e0567c9d4c09d9f9597c1274dfa3"
server_key_version: "encryptionkey@0"
I20260929 02:06:19.117002 15937 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 02:06:19.120422 15797 heartbeater.cc:503] Master 127.11.182.254:40823 was elected leader, sending a full tablet report...
I20260929 02:06:19.121428 15952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:19.123299 15937 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:19.123497 15937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "8c93f83496bc49cbb06285f187053ba7"
format_stamp: "Formatted at 2026-09-29 02:06:19 on dist-test-slave-q06p"
server_key: "27c394b3d5f8356912284e3fba95b901"
server_key_iv: "1a42e0567c9d4c09d9f9597c1274dfa3"
server_key_version: "encryptionkey@0"
I20260929 02:06:19.123729 15937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:19.174548 15937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:19.179313 15937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:19.179633 15937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:19.180806 15937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:19.182389 15937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:19.182471 15937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:19.182617 15937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:19.182678 15937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:19.219466 15937 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:37519
I20260929 02:06:19.219563 16064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:37519 every 8 connection(s)
I20260929 02:06:19.220830 15937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:19.230670 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 15937
I20260929 02:06:19.230918 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:19.231406 11995 external_mini_cluster.cc:1428] Setting key 0de9be99ffd21f433802641590bf932b
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[15563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647579, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:19.244325 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:19.223806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:47607 (local address 127.11.182.254:40823)
0929 02:06:19.224312 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:19.224317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:19.225175 (+ 858us) server_negotiation.cc:407] Connection header received
0929 02:06:19.227289 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:19.227296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:19.227400 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:19.227533 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:19.229419 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.230408 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.232554 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.233239 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.233392 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:19.237602 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:19.237607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:19.237610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:19.237645 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:19.240505 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.241728 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.241734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.241737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.241942 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.242412 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.242417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.242420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.242726 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:19.242941 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:19.243675 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 02:06:19.243908 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":272,"threads_started":1}
I20260929 02:06:19.245350 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:19.224215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40823 (local address 127.11.182.195:47607)
0929 02:06:19.224991 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:19.225040 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:19.226976 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:19.227813 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:19.227823 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:19.228482 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:19.229205 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:19.229235 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.230623 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.230628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:19.232340 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:19.232384 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.233556 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.233567 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:19.233944 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:19.234914 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:19.234954 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:19.237356 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:19.240695 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:19.240704 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:19.240720 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:19.241515 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:19.242076 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:19.242082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:19.242085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:19.242289 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:19.242883 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:19.242893 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:19.243298 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 02:06:19.243583 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 02:06:19.244169 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":295,"threads_started":1}
I20260929 02:06:19.247319 16065 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40823
I20260929 02:06:19.247677 16065 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:19.248571 16065 heartbeater.cc:511] Master 127.11.182.254:40823 requested a full tablet report, sending...
I20260929 02:06:19.250214 15607 ts_manager.cc:194] Registered new tserver with Master: 8c93f83496bc49cbb06285f187053ba7 (127.11.182.195:37519)
I20260929 02:06:19.251200 15607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:47607
I20260929 02:06:19.260870 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[15563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[15563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:06:19.314091 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:19.296532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.11.182.193:46819)
0929 02:06:19.297217 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:19.297224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:19.297249 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:06:19.297310 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:19.297316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:19.297529 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:19.297727 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:19.299263 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.300329 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.301641 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.302238 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.302391 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:19.307862 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:19.307879 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:19.307896 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:19.307942 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:19.311171 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.311956 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.311964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.311966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.312113 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.312542 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.312546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.312548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.312935 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:19.313184 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:19.313563 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 02:06:19.313801 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":346,"threads_started":1}
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[15563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:19.333798 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:19.319118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.11.182.254:40823)
0929 02:06:19.319375 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:19.319381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:19.319580 (+ 199us) server_negotiation.cc:407] Connection header received
0929 02:06:19.319783 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:19.319788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:19.319883 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:19.320016 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:19.321303 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.322059 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.323378 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.323782 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.323931 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:19.328132 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:19.328137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:19.328140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:19.328177 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:19.330931 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.331707 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.331711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.331714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.331869 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.332264 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.332267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.332269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.332589 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:19.332761 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:19.333204 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 02:06:19.333489 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260929 02:06:19.334421 15607 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42598
I20260929 02:06:19.335670 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15669
I20260929 02:06:19.349428 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:19.336619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.11.182.193:46819)
0929 02:06:19.336870 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:19.336876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:19.337104 (+ 228us) server_negotiation.cc:407] Connection header received
0929 02:06:19.337180 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:19.337191 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:19.337311 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:19.337399 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:19.338650 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.339542 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.340845 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.341314 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.341430 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:19.343948 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:19.343954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:19.343957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:19.343994 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:19.346665 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.347514 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.347519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.347522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.347647 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.348093 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.348097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.348100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.348482 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:19.348680 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:19.348954 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 02:06:19.349122 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:06:19.350122 15792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:19.434186 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 15669
I20260929 02:06:19.452961 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15803
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[15563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:06:19.471405 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:19.453957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.11.182.194:36685)
0929 02:06:19.454532 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:19.454539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:19.454563 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:06:19.454621 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:19.454627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:19.454921 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:19.455107 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:19.456513 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.457584 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.458942 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.459558 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.459716 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:19.464825 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:19.464844 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:19.464862 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:19.464919 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:19.468281 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.469063 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.469073 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.469076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.469222 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.469635 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.469663 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.469665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.470018 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:19.470324 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:19.470711 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 02:06:19.471032 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":310,"threads_started":1}
I20260929 02:06:19.472865 15926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:19.558120 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 15803
I20260929 02:06:19.574241 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15937
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[15563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:06:19.591980 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:19.575396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.11.182.195:37519)
0929 02:06:19.576066 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:19.576074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:19.576105 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:06:19.576168 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:19.576175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:19.576420 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:19.576558 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:19.577854 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.579204 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.580577 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:19.581200 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:19.581326 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:19.585731 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:19.585746 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:19.585760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:19.585810 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:19.589069 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.589893 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.589900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.589903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.590026 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:19.590479 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:19.590482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:19.590484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:19.590825 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:19.591127 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:19.591455 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 02:06:19.591695 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":348,"threads_started":1}
I20260929 02:06:19.593422 16060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:19.689219 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 15937
I20260929 02:06:19.705928 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 15576
I20260929 02:06:19.707257 15638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:19.809898 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 15576
2026-09-29T02:06:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4133 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[16085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[16085](info): set up 2 sockets
Sep 29 02:06:19 dist-test-slave-q06p krb5kdc[16085](info): commencing operation
krb5kdc: starting...
W20260929 02:06:20.919965 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:21 dist-test-slave-q06p krb5kdc[16085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:06:21.138453 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40855
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:21.496065 16098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:21.496388 16098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:21.496549 16098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:21.504052 16098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:21.504179 16098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:21.504251 16098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:21.504344 16098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:21.504451 16098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:21.514751 16098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:40855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:21.516178 16098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:21.517536 16098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:21.530115 16106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:21.530189 16104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:21.530360 16103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:21.531118 16098 server_base.cc:1034] running on GCE node
I20260929 02:06:21.532033 16098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:21.533634 16098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:21.534956 16098 hybrid_clock.cc:648] HybridClock initialized: now 1790647581534890 us; error 46 us; skew 500 ppm
I20260929 02:06:21.535333 16098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:21 dist-test-slave-q06p krb5kdc[16085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647581, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:21.538753 16098 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:06:21.540012 16098 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:21.540158 16098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:21.540506 16098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:21.543725 16098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "727ac07a335b4a11914971aac06588c8"
format_stamp: "Formatted at 2026-09-29 02:06:21 on dist-test-slave-q06p"
server_key: "f2f77ac98f7aef01ab09ef38debd67e1"
server_key_iv: "ad44152a3a01e716b37d3b02d8285db3"
server_key_version: "encryptionkey@0"
I20260929 02:06:21.544582 16098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "727ac07a335b4a11914971aac06588c8"
format_stamp: "Formatted at 2026-09-29 02:06:21 on dist-test-slave-q06p"
server_key: "f2f77ac98f7aef01ab09ef38debd67e1"
server_key_iv: "ad44152a3a01e716b37d3b02d8285db3"
server_key_version: "encryptionkey@0"
I20260929 02:06:21.550359 16098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:06:21.554522 16112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:21.556077 16098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:21.556320 16098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "727ac07a335b4a11914971aac06588c8"
format_stamp: "Formatted at 2026-09-29 02:06:21 on dist-test-slave-q06p"
server_key: "f2f77ac98f7aef01ab09ef38debd67e1"
server_key_iv: "ad44152a3a01e716b37d3b02d8285db3"
server_key_version: "encryptionkey@0"
I20260929 02:06:21.556572 16098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:21.587620 16098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:21.592073 16098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:21.592391 16098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:21.615927 16098 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:40855
I20260929 02:06:21.615950 16163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:40855 every 8 connection(s)
I20260929 02:06:21.616917 16098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:21.619421 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 16098
I20260929 02:06:21.619711 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:21.620539 11995 external_mini_cluster.cc:1428] Setting key d8dd50e3a550c52b8123c512f4974dcb
I20260929 02:06:21.621564 16164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:06:21 dist-test-slave-q06p krb5kdc[16085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:21.636981 16164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8: Bootstrap starting.
I20260929 02:06:21.640726 16164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:21.641947 16164 log.cc:824] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:21.644479 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:21.622644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.11.182.254:40855)
0929 02:06:21.623532 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:21.623545 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:21.623587 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:06:21.625112 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:21.625134 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:21.625568 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:21.625930 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:21.627830 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:21.629209 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:21.631092 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:21.631773 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:21.631937 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:21.636871 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:21.636880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:21.636897 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:21.636943 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:21.639994 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:21.640971 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:21.640979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:21.640985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:21.641151 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:21.641551 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:21.641554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:21.641556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:21.642164 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:21.642391 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:21.642985 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 02:06:21.643480 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":452,"threads_started":1}
I20260929 02:06:21.645201 16164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8: No bootstrap required, opened a new log
I20260929 02:06:21.652760 16164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ac07a335b4a11914971aac06588c8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40855 } }
I20260929 02:06:21.653182 16164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:21.653301 16164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727ac07a335b4a11914971aac06588c8, State: Initialized, Role: FOLLOWER
I20260929 02:06:21.653877 16164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ac07a335b4a11914971aac06588c8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40855 } }
I20260929 02:06:21.654126 16164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:21.654270 16164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:21.654480 16164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:21.656755 16164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ac07a335b4a11914971aac06588c8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40855 } }
I20260929 02:06:21.657226 16164 leader_election.cc:302] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 727ac07a335b4a11914971aac06588c8; no voters:
I20260929 02:06:21.657770 16164 leader_election.cc:288] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:21.657914 16169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:21.659229 16169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [term 1 LEADER]: Becoming Leader. State: Replica: 727ac07a335b4a11914971aac06588c8, State: Running, Role: LEADER
I20260929 02:06:21.659835 16169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ac07a335b4a11914971aac06588c8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40855 } }
I20260929 02:06:21.660218 16164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:21.664019 16170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "727ac07a335b4a11914971aac06588c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ac07a335b4a11914971aac06588c8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40855 } } }
I20260929 02:06:21.664526 16170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:21.665735 16171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 727ac07a335b4a11914971aac06588c8. Latest consensus state: current_term: 1 leader_uuid: "727ac07a335b4a11914971aac06588c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ac07a335b4a11914971aac06588c8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40855 } } }
I20260929 02:06:21.666040 16171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:21.666635 16178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:21.672508 16178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:21.682066 16178 catalog_manager.cc:1347] Generated new cluster ID: f534d0cab98a4699a615e7700924c1fc
I20260929 02:06:21.682221 16178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:21.701519 16178 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:21.703300 16178 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:21.719172 16178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8: Generated new TSK 0
I20260929 02:06:21.720145 16178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:06:21.779590 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40855
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:22.138595 16190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:22.139004 16190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:22.139179 16190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:22.147811 16190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:22.147994 16190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:22.148178 16190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:22.158721 16190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:22.160301 16190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:22.161891 16190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:22.175758 16196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:22.175784 16195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:22.176000 16198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:22.176609 16190 server_base.cc:1034] running on GCE node
I20260929 02:06:22.177297 16190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:22.178738 16190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:22.180197 16190 hybrid_clock.cc:648] HybridClock initialized: now 1790647582179943 us; error 90 us; skew 500 ppm
I20260929 02:06:22.180598 16190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:22 dist-test-slave-q06p krb5kdc[16085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647582, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:22.184055 16190 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:06:22.185323 16190 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:22.185462 16190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:22.185811 16190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:22.188951 16190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "220946d9d8dd499d80102c9dd91a9f26"
format_stamp: "Formatted at 2026-09-29 02:06:22 on dist-test-slave-q06p"
server_key: "5979eef28d7ad9b26d4796c4f264debc"
server_key_iv: "ed1ea2b4f1d8345273963dc1044c9eb9"
server_key_version: "encryptionkey@0"
I20260929 02:06:22.189798 16190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "220946d9d8dd499d80102c9dd91a9f26"
format_stamp: "Formatted at 2026-09-29 02:06:22 on dist-test-slave-q06p"
server_key: "5979eef28d7ad9b26d4796c4f264debc"
server_key_iv: "ed1ea2b4f1d8345273963dc1044c9eb9"
server_key_version: "encryptionkey@0"
I20260929 02:06:22.195391 16190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 02:06:22.200007 16204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:22.201756 16190 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:06:22.201967 16190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "220946d9d8dd499d80102c9dd91a9f26"
format_stamp: "Formatted at 2026-09-29 02:06:22 on dist-test-slave-q06p"
server_key: "5979eef28d7ad9b26d4796c4f264debc"
server_key_iv: "ed1ea2b4f1d8345273963dc1044c9eb9"
server_key_version: "encryptionkey@0"
I20260929 02:06:22.202214 16190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:22.226257 16190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:22.231269 16190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:22.231570 16190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:22.232681 16190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:22.234156 16190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:22.234287 16190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:22.234429 16190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:22.234525 16190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:22.268224 16190 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:39799
I20260929 02:06:22.268309 16316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:39799 every 8 connection(s)
I20260929 02:06:22.269110 16190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:22.272861 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 16190
I20260929 02:06:22.273094 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:22.273916 11995 external_mini_cluster.cc:1428] Setting key 7353c4d8a750f398476dbceed84ef496
Sep 29 02:06:22 dist-test-slave-q06p krb5kdc[16085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647582, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:22.290643 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:22.271634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:40863 (local address 127.11.182.254:40855)
0929 02:06:22.272071 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:22.272075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:22.273120 (+ 1045us) server_negotiation.cc:407] Connection header received
0929 02:06:22.275424 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:22.275430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:22.275499 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:22.275615 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:22.277683 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:22.278399 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:22.281077 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:22.281482 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:22.281586 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:22.285086 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:22.285091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:22.285094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:22.285125 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:22.287674 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:22.288403 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:22.288413 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:22.288415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:22.288526 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:22.288947 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:22.288952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:22.288955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:22.289205 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:22.289362 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:22.290062 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 02:06:22.290278 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":241,"threads_started":1}
I20260929 02:06:22.291746 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:22.272098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40855 (local address 127.11.182.193:40863)
0929 02:06:22.272917 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:22.272984 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:22.275168 (+ 2184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:22.275881 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:22.275890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:22.276632 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:22.277491 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:22.277520 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:22.278562 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:22.278570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:22.280822 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:22.280850 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:22.281716 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:22.281725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:22.282019 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:22.282712 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:22.282734 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:22.284861 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:22.287817 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:22.287827 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:22.287853 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:22.288268 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:22.288632 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:22.288638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:22.288642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:22.288818 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:22.289331 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:22.289339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:22.289680 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 02:06:22.289950 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 02:06:22.290480 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":303,"threads_started":1}
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:06:22.293418 16317 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40855
I20260929 02:06:22.293743 16317 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:22.294395 16317 heartbeater.cc:511] Master 127.11.182.254:40855 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
I20260929 02:06:22.296593 16129 ts_manager.cc:194] Registered new tserver with Master: 220946d9d8dd499d80102c9dd91a9f26 (127.11.182.193:39799)
I20260929 02:06:22.298074 16129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:40863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:06:22.319015 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40855
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:22.647159 16323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:22.647461 16323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:22.647563 16323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:22.654953 16323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:22.655074 16323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:22.655191 16323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:22.665212 16323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:22.666687 16323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:22.668054 16323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:22.681305 16331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:22.681244 16329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:22.681244 16328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:22.682206 16323 server_base.cc:1034] running on GCE node
I20260929 02:06:22.682981 16323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:22.684003 16323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:22.685271 16323 hybrid_clock.cc:648] HybridClock initialized: now 1790647582685202 us; error 53 us; skew 500 ppm
I20260929 02:06:22.685628 16323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:22 dist-test-slave-q06p krb5kdc[16085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647582, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:22.693936 16323 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:06:22.695293 16323 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:22.695436 16323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:22.695777 16323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:22.698935 16323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "5bde98a0fe884878bf03e4391902275a"
format_stamp: "Formatted at 2026-09-29 02:06:22 on dist-test-slave-q06p"
server_key: "3be0c1907dc05e8ecceb57629dc8772e"
server_key_iv: "eeaeec2aca232cb554f4d173560abe18"
server_key_version: "encryptionkey@0"
I20260929 02:06:22.700440 16323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "5bde98a0fe884878bf03e4391902275a"
format_stamp: "Formatted at 2026-09-29 02:06:22 on dist-test-slave-q06p"
server_key: "3be0c1907dc05e8ecceb57629dc8772e"
server_key_iv: "eeaeec2aca232cb554f4d173560abe18"
server_key_version: "encryptionkey@0"
I20260929 02:06:22.706588 16323 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 02:06:22.710614 16337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:22.712395 16323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:22.712623 16323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "5bde98a0fe884878bf03e4391902275a"
format_stamp: "Formatted at 2026-09-29 02:06:22 on dist-test-slave-q06p"
server_key: "3be0c1907dc05e8ecceb57629dc8772e"
server_key_iv: "eeaeec2aca232cb554f4d173560abe18"
server_key_version: "encryptionkey@0"
I20260929 02:06:22.712885 16323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:22.728189 16323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:22.732767 16323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:22.733075 16323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:22.734076 16323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:22.735507 16323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:22.735641 16323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:22.735771 16323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:22.735859 16323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:22.771289 16323 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:40403
I20260929 02:06:22.771358 16449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:40403 every 8 connection(s)
I20260929 02:06:22.772290 16323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:22.779050 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 16323
I20260929 02:06:22.779260 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:22.779757 11995 external_mini_cluster.cc:1428] Setting key 11caebba57ea74a4e6c17d48b7e25d04
Sep 29 02:06:22 dist-test-slave-q06p krb5kdc[16085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647582, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:22.794037 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:22.775107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:36013 (local address 127.11.182.254:40855)
0929 02:06:22.775282 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:22.775287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:22.776482 (+ 1195us) server_negotiation.cc:407] Connection header received
0929 02:06:22.778395 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:22.778401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:22.778494 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:22.778651 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:22.780687 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:22.781413 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:22.783096 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:22.783645 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:22.783819 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:22.787714 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:22.787720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:22.787723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:22.787759 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:22.790457 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:22.791441 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:22.791445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:22.791448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:22.791585 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:22.792062 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:22.792067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:22.792069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:22.792493 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:22.792652 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:22.793427 (+ 775us) server_negotiation.cc:299] Negotiation successful
0929 02:06:22.793665 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:06:22.795178 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:22.775544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40855 (local address 127.11.182.194:36013)
0929 02:06:22.776302 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:22.776341 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:22.778090 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:22.778991 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:22.779002 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:22.779693 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:22.780464 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:22.780494 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:22.781568 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:22.781573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:22.782947 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:22.782960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:22.783947 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:22.783956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:22.784211 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:22.785003 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:22.785031 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:22.787415 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:22.790659 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:22.790670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:22.790691 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:22.791284 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:22.791706 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:22.791712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:22.791715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:22.791927 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:22.792629 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:22.792638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:22.793038 (+ 400us) client_negotiation.cc:769] Sending connection context
0929 02:06:22.793321 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 02:06:22.793926 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":313,"threads_started":1}
I20260929 02:06:22.796761 16450 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40855
I20260929 02:06:22.797109 16450 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:06:22.797847 16450 heartbeater.cc:511] Master 127.11.182.254:40855 requested a full tablet report, sending...
I20260929 02:06:22.799500 16129 ts_manager.cc:194] Registered new tserver with Master: 5bde98a0fe884878bf03e4391902275a (127.11.182.194:40403)
I20260929 02:06:22.800648 16129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:36013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:06:22.831240 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40855
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:23.175374 16455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:23.175689 16455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:23.175797 16455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:23.183284 16455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:23.183424 16455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:23.183542 16455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:23.193619 16455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:23.194973 16455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:23.196321 16455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:23.209945 16460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:23.210041 16461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:23.210062 16463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:23.211167 16455 server_base.cc:1034] running on GCE node
I20260929 02:06:23.211907 16455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:23.213115 16455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:23.214391 16455 hybrid_clock.cc:648] HybridClock initialized: now 1790647583214316 us; error 68 us; skew 500 ppm
I20260929 02:06:23.214905 16455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:23 dist-test-slave-q06p krb5kdc[16085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647583, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:23.218736 16455 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:06:23.220342 16455 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:23.220450 16455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:23.220804 16455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:23.224668 16455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "d35f181b2c134107a6d3d136f0a7427f"
format_stamp: "Formatted at 2026-09-29 02:06:23 on dist-test-slave-q06p"
server_key: "67b4ed5ba9175242703edecf3cbc92ce"
server_key_iv: "05b9bcd94b656ef79f9b8a07b561a4dd"
server_key_version: "encryptionkey@0"
I20260929 02:06:23.225621 16455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "d35f181b2c134107a6d3d136f0a7427f"
format_stamp: "Formatted at 2026-09-29 02:06:23 on dist-test-slave-q06p"
server_key: "67b4ed5ba9175242703edecf3cbc92ce"
server_key_iv: "05b9bcd94b656ef79f9b8a07b561a4dd"
server_key_version: "encryptionkey@0"
I20260929 02:06:23.232555 16455 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 02:06:23.236907 16469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:23.239310 16455 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:23.239531 16455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "d35f181b2c134107a6d3d136f0a7427f"
format_stamp: "Formatted at 2026-09-29 02:06:23 on dist-test-slave-q06p"
server_key: "67b4ed5ba9175242703edecf3cbc92ce"
server_key_iv: "05b9bcd94b656ef79f9b8a07b561a4dd"
server_key_version: "encryptionkey@0"
I20260929 02:06:23.239773 16455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:23.289516 16455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:23.294400 16455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:23.294721 16455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:23.295888 16455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:23.297407 16455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:23.297495 16455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:23.297570 16455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:23.297624 16455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:23.301457 16317 heartbeater.cc:503] Master 127.11.182.254:40855 was elected leader, sending a full tablet report...
I20260929 02:06:23.333454 16455 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:45651
I20260929 02:06:23.333546 16581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:45651 every 8 connection(s)
I20260929 02:06:23.334399 16455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:23.343338 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 16455
I20260929 02:06:23.343595 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:23.344087 11995 external_mini_cluster.cc:1428] Setting key 4d9ec771833d78685a14f4e51696b8e4
Sep 29 02:06:23 dist-test-slave-q06p krb5kdc[16085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647583, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:23.356690 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.336986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:36675 (local address 127.11.182.254:40855)
0929 02:06:23.337451 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.337456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.338508 (+ 1052us) server_negotiation.cc:407] Connection header received
0929 02:06:23.340472 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.340476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.340557 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.340659 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:23.342302 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.343309 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.345540 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.345951 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.346094 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.350058 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:23.350063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:23.350066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:23.350102 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:23.353105 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:23.354011 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:23.354017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:23.354021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:23.354163 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:23.354653 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:23.354664 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:23.354666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:23.355028 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:23.355200 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.356010 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.356254 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260929 02:06:23.357774 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.337606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40855 (local address 127.11.182.195:36675)
0929 02:06:23.338341 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.338390 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:23.340157 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:23.340874 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:23.340883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:23.341438 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:23.342065 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.342083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.343545 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.343551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.345380 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.345404 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.346200 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.346207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.346453 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.347361 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:23.347390 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:23.349805 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:23.353327 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:23.353340 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:23.353363 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:23.353856 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:23.354296 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:23.354302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:23.354306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:23.354512 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:23.355173 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:23.355182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:23.355598 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 02:06:23.355899 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 02:06:23.356520 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":285,"threads_started":1}
I20260929 02:06:23.359550 16582 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40855
I20260929 02:06:23.359934 16582 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:23.360795 16582 heartbeater.cc:511] Master 127.11.182.254:40855 requested a full tablet report, sending...
I20260929 02:06:23.362514 16128 ts_manager.cc:194] Registered new tserver with Master: d35f181b2c134107a6d3d136f0a7427f (127.11.182.195:45651)
I20260929 02:06:23.363543 16128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:36675
I20260929 02:06:23.373109 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:06:23.391058 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.377973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.11.182.254:40855)
0929 02:06:23.378156 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.378160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.378581 (+ 421us) server_negotiation.cc:407] Connection header received
0929 02:06:23.378763 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.378768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.378918 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.379022 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:23.380494 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.381237 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.382556 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.383070 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.383170 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.385541 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:23.385546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:23.385549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:23.385583 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:23.388453 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:23.389169 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:23.389174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:23.389176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:23.389301 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:23.389624 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:23.389627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:23.389629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:23.389986 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:23.390164 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.390506 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.390713 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:06:23.395771 16128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:23.402705 16128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:23.426384 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.417973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45651 (local address 127.0.0.1:37274)
0929 02:06:23.418547 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.418567 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:23.418675 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:23.419751 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:23.419759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:23.419782 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:23.420168 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.420177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.421673 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.421680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.424000 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.424013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.425307 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.425315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.425501 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.425551 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 02:06:23.425693 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 02:06:23.425936 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":48,"thread_start_us":302,"threads_started":1}
I20260929 02:06:23.426718 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.418230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.11.182.195:45651)
0929 02:06:23.418918 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.418927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.418956 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:06:23.419046 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.419053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.419270 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.419490 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:23.420392 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.421539 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.424167 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.425105 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.425334 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.425520 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.425979 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.426171 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":297,"threads_started":1}
I20260929 02:06:23.427063 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.415456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.11.182.194:40403)
0929 02:06:23.416116 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.416127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.416896 (+ 769us) server_negotiation.cc:407] Connection header received
0929 02:06:23.417214 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.417224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.417490 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.417651 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:23.419432 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.421073 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.423869 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.425312 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.425496 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.425684 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.426303 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.426596 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":323,"threads_started":1}
I20260929 02:06:23.427139 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.416312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.0.0.1:33278)
0929 02:06:23.417858 (+ 1546us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.417879 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:23.418009 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:23.418729 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:23.418735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:23.418768 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:23.419286 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.419303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.421320 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.421325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.423944 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.423959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.425739 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.425745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.425882 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.426055 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 02:06:23.426140 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 02:06:23.426337 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1407,"mutex_wait_us":157,"spinlock_wait_cycles":10112,"thread_start_us":1267,"threads_started":1}
I20260929 02:06:23.428216 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.416152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.11.182.193:39799)
0929 02:06:23.417509 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.417519 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.418149 (+ 630us) server_negotiation.cc:407] Connection header received
0929 02:06:23.418251 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.418257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.418514 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.419520 (+ 1006us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:23.419737 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.421190 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.424109 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.425499 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.427464 (+ 1965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.427649 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.427741 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.427913 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1204,"thread_start_us":973,"threads_started":1}
I20260929 02:06:23.428539 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.415397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:40403 (local address 127.0.0.1:38750)
0929 02:06:23.416720 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.416767 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:23.416962 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:23.417936 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:23.417942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:23.417997 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:23.419179 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.419195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.421218 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.421223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.423706 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.423721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.425688 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.425695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.425869 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.426059 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 02:06:23.426142 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 02:06:23.428182 (+ 2040us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":2168,"spinlock_wait_cycles":15616,"thread_start_us":1093,"threads_started":1}
I20260929 02:06:23.430787 16517 tablet_service.cc:1467] Processing CreateTablet for tablet 43866d7d7fb54c12b5d7566add632b9a (DEFAULT_TABLE table=test-table [id=94b04c838011436e8a7ed01795882d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:23.432358 16517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43866d7d7fb54c12b5d7566add632b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:23.431938 16385 tablet_service.cc:1467] Processing CreateTablet for tablet 43866d7d7fb54c12b5d7566add632b9a (DEFAULT_TABLE table=test-table [id=94b04c838011436e8a7ed01795882d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:23.433214 16385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43866d7d7fb54c12b5d7566add632b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:23.433701 16252 tablet_service.cc:1467] Processing CreateTablet for tablet 43866d7d7fb54c12b5d7566add632b9a (DEFAULT_TABLE table=test-table [id=94b04c838011436e8a7ed01795882d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:23.434641 16252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43866d7d7fb54c12b5d7566add632b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:23.447290 16602 tablet_bootstrap.cc:492] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: Bootstrap starting.
I20260929 02:06:23.451745 16601 tablet_bootstrap.cc:492] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: Bootstrap starting.
I20260929 02:06:23.452052 16602 tablet_bootstrap.cc:654] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:23.453172 16603 tablet_bootstrap.cc:492] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: Bootstrap starting.
I20260929 02:06:23.453612 16602 log.cc:824] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:23.455449 16601 tablet_bootstrap.cc:654] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:23.456449 16601 log.cc:824] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:23.457307 16603 tablet_bootstrap.cc:654] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:23.458896 16603 log.cc:824] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:23.459613 16601 tablet_bootstrap.cc:492] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: No bootstrap required, opened a new log
I20260929 02:06:23.459913 16601 ts_tablet_manager.cc:1397] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 02:06:23.461571 16602 tablet_bootstrap.cc:492] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: No bootstrap required, opened a new log
I20260929 02:06:23.461866 16602 ts_tablet_manager.cc:1397] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260929 02:06:23.462990 16603 tablet_bootstrap.cc:492] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: No bootstrap required, opened a new log
I20260929 02:06:23.463297 16603 ts_tablet_manager.cc:1397] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260929 02:06:23.469822 16601 raft_consensus.cc:348] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } }
I20260929 02:06:23.470237 16601 raft_consensus.cc:374] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:23.470332 16601 raft_consensus.cc:729] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220946d9d8dd499d80102c9dd91a9f26, State: Initialized, Role: FOLLOWER
I20260929 02:06:23.471032 16601 consensus_queue.cc:260] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } }
I20260929 02:06:23.472666 16601 ts_tablet_manager.cc:1428] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260929 02:06:23.473884 16603 raft_consensus.cc:348] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } }
I20260929 02:06:23.473896 16602 raft_consensus.cc:348] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } }
I20260929 02:06:23.474334 16603 raft_consensus.cc:374] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:23.474321 16602 raft_consensus.cc:374] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:23.474423 16603 raft_consensus.cc:729] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35f181b2c134107a6d3d136f0a7427f, State: Initialized, Role: FOLLOWER
I20260929 02:06:23.474442 16602 raft_consensus.cc:729] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bde98a0fe884878bf03e4391902275a, State: Initialized, Role: FOLLOWER
I20260929 02:06:23.475090 16602 consensus_queue.cc:260] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } }
I20260929 02:06:23.475090 16603 consensus_queue.cc:260] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } }
I20260929 02:06:23.476984 16450 heartbeater.cc:503] Master 127.11.182.254:40855 was elected leader, sending a full tablet report...
I20260929 02:06:23.477059 16582 heartbeater.cc:503] Master 127.11.182.254:40855 was elected leader, sending a full tablet report...
I20260929 02:06:23.477717 16603 ts_tablet_manager.cc:1428] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260929 02:06:23.478786 16602 ts_tablet_manager.cc:1428] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260929 02:06:23.521576 16318 tablet.cc:2369] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:06:23.524621 16451 tablet.cc:2369] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:06:23.580955 16609 raft_consensus.cc:482] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:23.581271 16609 raft_consensus.cc:504] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } }
I20260929 02:06:23.583074 16609 leader_election.cc:288] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 220946d9d8dd499d80102c9dd91a9f26 (127.11.182.193:39799), d35f181b2c134107a6d3d136f0a7427f (127.11.182.195:45651)
W20260929 02:06:23.586026 16583 tablet.cc:2369] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:06:23.590546 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.583507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:44789 (local address 127.11.182.193:39799)
0929 02:06:23.583738 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.583743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.584491 (+ 748us) server_negotiation.cc:407] Connection header received
0929 02:06:23.584594 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.584600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.584717 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.584842 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:23.585768 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.586719 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.588727 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.589536 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.589735 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.589820 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.590142 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.590275 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:06:23.590708 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.583459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.11.182.194:44789)
0929 02:06:23.584224 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.584245 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:23.584393 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:23.585060 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:23.585070 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:23.585100 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:23.585569 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.585585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.586899 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.586909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.588507 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.588519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.589784 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.589791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.589960 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.589985 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:06:23.590112 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 02:06:23.590249 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":429,"threads_started":1}
I20260929 02:06:23.590761 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.584418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45651 (local address 127.11.182.194:57217)
0929 02:06:23.585268 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.585293 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:23.585446 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:23.585777 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:23.585785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:23.585813 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:23.586239 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.586255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.587475 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.587481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.588859 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:23.588869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.589968 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.589975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:23.590147 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.590169 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:06:23.590269 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:06:23.590406 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":485,"threads_started":1}
I20260929 02:06:23.591046 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.584693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:57217 (local address 127.11.182.195:45651)
0929 02:06:23.584963 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.584970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.585393 (+ 423us) server_negotiation.cc:407] Connection header received
0929 02:06:23.585576 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.585580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.585634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.585737 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:23.586460 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.587334 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.589007 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.589751 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.589945 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.590015 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.590304 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.590424 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:06:23.591809 16272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43866d7d7fb54c12b5d7566add632b9a" candidate_uuid: "5bde98a0fe884878bf03e4391902275a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220946d9d8dd499d80102c9dd91a9f26" is_pre_election: true
I20260929 02:06:23.591857 16537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43866d7d7fb54c12b5d7566add632b9a" candidate_uuid: "5bde98a0fe884878bf03e4391902275a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35f181b2c134107a6d3d136f0a7427f" is_pre_election: true
I20260929 02:06:23.592339 16537 raft_consensus.cc:2454] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bde98a0fe884878bf03e4391902275a in term 0.
I20260929 02:06:23.592338 16272 raft_consensus.cc:2454] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bde98a0fe884878bf03e4391902275a in term 0.
I20260929 02:06:23.593034 16340 leader_election.cc:302] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bde98a0fe884878bf03e4391902275a, d35f181b2c134107a6d3d136f0a7427f; no voters:
I20260929 02:06:23.593433 16609 raft_consensus.cc:2781] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:23.593559 16609 raft_consensus.cc:482] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:23.593703 16609 raft_consensus.cc:3037] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:23.595944 16609 raft_consensus.cc:504] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } }
I20260929 02:06:23.596529 16609 leader_election.cc:288] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [CANDIDATE]: Term 1 election: Requested vote from peers 220946d9d8dd499d80102c9dd91a9f26 (127.11.182.193:39799), d35f181b2c134107a6d3d136f0a7427f (127.11.182.195:45651)
I20260929 02:06:23.597108 16272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43866d7d7fb54c12b5d7566add632b9a" candidate_uuid: "5bde98a0fe884878bf03e4391902275a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220946d9d8dd499d80102c9dd91a9f26"
I20260929 02:06:23.597214 16537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43866d7d7fb54c12b5d7566add632b9a" candidate_uuid: "5bde98a0fe884878bf03e4391902275a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35f181b2c134107a6d3d136f0a7427f"
I20260929 02:06:23.597321 16272 raft_consensus.cc:3037] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:23.597491 16537 raft_consensus.cc:3037] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:23.600994 16272 raft_consensus.cc:2454] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bde98a0fe884878bf03e4391902275a in term 1.
I20260929 02:06:23.601269 16537 raft_consensus.cc:2454] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bde98a0fe884878bf03e4391902275a in term 1.
I20260929 02:06:23.601593 16340 leader_election.cc:302] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220946d9d8dd499d80102c9dd91a9f26, 5bde98a0fe884878bf03e4391902275a; no voters:
I20260929 02:06:23.601969 16609 raft_consensus.cc:2781] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:23.602648 16609 raft_consensus.cc:686] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 1 LEADER]: Becoming Leader. State: Replica: 5bde98a0fe884878bf03e4391902275a, State: Running, Role: LEADER
I20260929 02:06:23.603266 16609 consensus_queue.cc:237] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } }
I20260929 02:06:23.615015 16127 catalog_manager.cc:5495] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bde98a0fe884878bf03e4391902275a (127.11.182.194). New cstate: current_term: 1 leader_uuid: "5bde98a0fe884878bf03e4391902275a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bde98a0fe884878bf03e4391902275a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 } health_report { overall_health: UNKNOWN } } }
I20260929 02:06:23.671257 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.665924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.11.182.194:40403)
0929 02:06:23.666141 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.666147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.666165 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:06:23.666258 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.666262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.666339 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.666520 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:23.666904 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.667601 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.669121 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.669525 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.669653 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.670127 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:23.670636 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:23.670792 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.670909 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.670988 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:06:23.679586 16537 raft_consensus.cc:1261] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [term 1 FOLLOWER]: Refusing update from remote peer 5bde98a0fe884878bf03e4391902275a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:23.679627 16272 raft_consensus.cc:1261] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [term 1 FOLLOWER]: Refusing update from remote peer 5bde98a0fe884878bf03e4391902275a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:23.680545 16612 consensus_queue.cc:1035] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [LEADER]: Connected to new peer: Peer: permanent_uuid: "220946d9d8dd499d80102c9dd91a9f26" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 39799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:23.680872 16609 consensus_queue.cc:1035] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35f181b2c134107a6d3d136f0a7427f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:23.692538 16615 mvcc.cc:205] Tried to move back new op lower bound from 7334492502742048768 to 7334492502443593728. Current Snapshot: MvccSnapshot[applied={T|T < 7334492502742048768}]
I20260929 02:06:23.705646 16127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:23.705914 16127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:23.709745 16127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 727ac07a335b4a11914971aac06588c8: Sending DeleteTablet for 3 replicas of tablet 43866d7d7fb54c12b5d7566add632b9a
I20260929 02:06:23.710899 16385 tablet_service.cc:1514] Processing DeleteTablet for tablet 43866d7d7fb54c12b5d7566add632b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:23 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:38750
I20260929 02:06:23.711028 16517 tablet_service.cc:1514] Processing DeleteTablet for tablet 43866d7d7fb54c12b5d7566add632b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:23 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:37274
I20260929 02:06:23.711107 16252 tablet_service.cc:1514] Processing DeleteTablet for tablet 43866d7d7fb54c12b5d7566add632b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:23 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:33278
I20260929 02:06:23.712327 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 16190
I20260929 02:06:23.727700 16626 tablet_replica.cc:330] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: stopping tablet replica
I20260929 02:06:23.728168 16626 raft_consensus.cc:2229] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:23.728704 16626 raft_consensus.cc:2258] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:23.732431 16625 tablet_replica.cc:330] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: stopping tablet replica
I20260929 02:06:23.732825 16627 tablet_replica.cc:330] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: stopping tablet replica
I20260929 02:06:23.733250 16627 raft_consensus.cc:2229] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:23.733738 16627 raft_consensus.cc:2258] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:23.733964 16626 ts_tablet_manager.cc:1905] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:23.735855 16627 ts_tablet_manager.cc:1905] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:06:23 dist-test-slave-q06p krb5kdc[16085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:06:23.741529 16626 ts_tablet_manager.cc:1918] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:23.741708 16626 log.cc:1196] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/43866d7d7fb54c12b5d7566add632b9a
I20260929 02:06:23.742254 16626 ts_tablet_manager.cc:1939] T 43866d7d7fb54c12b5d7566add632b9a P d35f181b2c134107a6d3d136f0a7427f: Deleting consensus metadata
I20260929 02:06:23.742969 16627 ts_tablet_manager.cc:1918] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:23.743124 16627 log.cc:1196] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/43866d7d7fb54c12b5d7566add632b9a
I20260929 02:06:23.743299 16625 raft_consensus.cc:2229] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:23.743621 16627 ts_tablet_manager.cc:1939] T 43866d7d7fb54c12b5d7566add632b9a P 220946d9d8dd499d80102c9dd91a9f26: Deleting consensus metadata
I20260929 02:06:23.744062 16625 raft_consensus.cc:2258] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:23.744253 16115 catalog_manager.cc:4841] TS d35f181b2c134107a6d3d136f0a7427f (127.11.182.195:45651): tablet 43866d7d7fb54c12b5d7566add632b9a (table test-table [id=94b04c838011436e8a7ed01795882d4d]) successfully deleted
I20260929 02:06:23.746356 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.714177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.11.182.193:39799)
0929 02:06:23.714407 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.714415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.724775 (+ 10360us) server_negotiation.cc:407] Connection header received
0929 02:06:23.725000 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.725008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.725130 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.725257 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:23.727475 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.728616 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.730687 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.731427 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.731550 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.738199 (+ 6649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:23.738214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:23.738233 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:23.738280 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:23.742670 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:23.743890 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:23.743899 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:23.743902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:23.744092 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:23.744733 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:23.744738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:23.744743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:23.745214 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:23.745519 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.745785 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.745986 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:06:23.746312 16625 ts_tablet_manager.cc:1905] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:23.747031 16312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:23.754536 16625 ts_tablet_manager.cc:1918] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:23.754717 16625 log.cc:1196] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/43866d7d7fb54c12b5d7566add632b9a
I20260929 02:06:23.755299 16625 ts_tablet_manager.cc:1939] T 43866d7d7fb54c12b5d7566add632b9a P 5bde98a0fe884878bf03e4391902275a: Deleting consensus metadata
I20260929 02:06:23.757052 16116 catalog_manager.cc:4841] TS 5bde98a0fe884878bf03e4391902275a (127.11.182.194:40403): tablet 43866d7d7fb54c12b5d7566add632b9a (table test-table [id=94b04c838011436e8a7ed01795882d4d]) successfully deleted
I20260929 02:06:23.853554 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 16190
W20260929 02:06:23.872246 16115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.182.193:39799 (error 108)
I20260929 02:06:23.872963 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 16323
I20260929 02:06:23.874668 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.873960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.0.0.1:33296)
0929 02:06:23.874240 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.874474 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260929 02:06:23.875016 16115 catalog_manager.cc:4584] TS 220946d9d8dd499d80102c9dd91a9f26 (127.11.182.193:39799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43866d7d7fb54c12b5d7566add632b9a: Network error: Client connection negotiation failed: client connection to 127.11.182.193:39799: connect: Connection refused (error 111)
Sep 29 02:06:23 dist-test-slave-q06p krb5kdc[16085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:06:23.892984 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.873795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.11.182.194:40403)
0929 02:06:23.873994 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:23.874001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:23.874120 (+ 119us) server_negotiation.cc:407] Connection header received
0929 02:06:23.874295 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:23.874303 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:23.874519 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:23.874660 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:23.876478 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.877531 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.879507 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:23.880006 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:23.880134 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:23.885492 (+ 5358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:23.885510 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:23.885526 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:23.885608 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:23.889543 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:23.890536 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:23.890545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:23.890547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:23.890718 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:23.891185 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:23.891189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:23.891191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:23.891678 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:23.891977 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:23.892325 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 02:06:23.892512 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:06:23.893736 16445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:23.928772 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.928281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.0.0.1:33298)
0929 02:06:23.928489 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.928590 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:06:23.929848 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.929471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.0.0.1:33308)
0929 02:06:23.929657 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.929733 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 02:06:23.990650 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.990168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.0.0.1:33316)
0929 02:06:23.990401 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.990487 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 02:06:23.991639 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:23.991304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.0.0.1:33332)
0929 02:06:23.991449 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:23.991524 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:06:24.011974 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 16323
I20260929 02:06:24.031270 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 16455
Sep 29 02:06:24 dist-test-slave-q06p krb5kdc[16085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:06:24.047497 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:24.032126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.11.182.195:45651)
0929 02:06:24.032351 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:24.032357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:24.032379 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:06:24.032578 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:24.032582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:24.032671 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:24.032795 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:24.034081 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:24.034823 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:24.036136 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:24.036525 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:24.036676 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:24.041228 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:24.041245 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:24.041262 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:24.041302 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:24.044433 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:24.045281 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:24.045290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:24.045293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:24.045455 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:24.045868 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:24.045872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:24.045874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:24.046252 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:24.046496 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:24.046983 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 02:06:24.047211 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 02:06:24.048219 16577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:24.072213 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:24.071741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.0.0.1:33338)
0929 02:06:24.071959 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:24.072050 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 02:06:24.073307 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:24.072912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.0.0.1:33354)
0929 02:06:24.073091 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:24.073196 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 02:06:24.138738 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 16455
I20260929 02:06:24.155019 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 16098
I20260929 02:06:24.156388 16159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:24.264612 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:24.263937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:39799 (local address 127.0.0.1:33368)
0929 02:06:24.264254 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:24.264368 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 02:06:24.264932 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 16098
2026-09-29T02:06:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4453 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T02:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:24Z Disabled control of system clock
I20260929 02:06:24.332043 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40643
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:44641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:06:24.669219 16637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:24.669569 16637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:24.669792 16637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:24.677672 16637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:24.677802 16637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:24.677870 16637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:24.677953 16637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:24.678012 16637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:24.687755 16637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:40643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:24.689165 16637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:24.691087 16637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:24.704090 16642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:24.704046 16645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:24.703995 16643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:24.704998 16637 server_base.cc:1034] running on GCE node
I20260929 02:06:24.705976 16637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:24.707687 16637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:24.708942 16637 hybrid_clock.cc:648] HybridClock initialized: now 1790647584708896 us; error 49 us; skew 500 ppm
I20260929 02:06:24.709342 16637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:06:24.711776 16637 webserver.cc:460] Webserver started at http://127.11.182.254:43993/ using document root <none> and password file <none>
I20260929 02:06:24.712563 16637 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:24.712700 16637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:24.713061 16637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:24.716161 16637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "9664a5f408874fe887c0e5cb33888750"
format_stamp: "Formatted at 2026-09-29 02:06:24 on dist-test-slave-q06p"
server_key: "5990fad4309c6d4e7f365f1503226c7b"
server_key_iv: "2c665f8f6e6b33c3d3d203bfe58b5b77"
server_key_version: "encryptionkey@0"
I20260929 02:06:24.716987 16637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "9664a5f408874fe887c0e5cb33888750"
format_stamp: "Formatted at 2026-09-29 02:06:24 on dist-test-slave-q06p"
server_key: "5990fad4309c6d4e7f365f1503226c7b"
server_key_iv: "2c665f8f6e6b33c3d3d203bfe58b5b77"
server_key_version: "encryptionkey@0"
I20260929 02:06:24.722673 16637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:06:24.726411 16651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:24.728048 16637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:24.728258 16637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "9664a5f408874fe887c0e5cb33888750"
format_stamp: "Formatted at 2026-09-29 02:06:24 on dist-test-slave-q06p"
server_key: "5990fad4309c6d4e7f365f1503226c7b"
server_key_iv: "2c665f8f6e6b33c3d3d203bfe58b5b77"
server_key_version: "encryptionkey@0"
I20260929 02:06:24.728523 16637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:24.735142 16637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:24.736037 16637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:24.736358 16637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:24.755544 16637 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:40643
I20260929 02:06:24.755577 16702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:40643 every 8 connection(s)
I20260929 02:06:24.756984 16637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:24.760835 16703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:24.761837 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 16637
I20260929 02:06:24.762024 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:24.762764 11995 external_mini_cluster.cc:1428] Setting key 73bad0fe1ab64764551c753f29084651
I20260929 02:06:24.771422 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:24.764907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.11.182.254:40643)
0929 02:06:24.765958 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:24.765970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:24.766013 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:06:24.767575 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:24.767603 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:24.768056 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:24.768472 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:24.769533 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:24.769539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:24.769552 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:24.769567 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:24.769625 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:24.769640 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:24.769881 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:24.770026 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 02:06:24.770334 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":361,"threads_started":1}
I20260929 02:06:24.774894 16703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750: Bootstrap starting.
I20260929 02:06:24.778043 16703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:24.779198 16703 log.cc:824] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:24.782332 16703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750: No bootstrap required, opened a new log
I20260929 02:06:24.789870 16703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9664a5f408874fe887c0e5cb33888750" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40643 } }
I20260929 02:06:24.790308 16703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:24.790421 16703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9664a5f408874fe887c0e5cb33888750, State: Initialized, Role: FOLLOWER
I20260929 02:06:24.791083 16703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9664a5f408874fe887c0e5cb33888750" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40643 } }
I20260929 02:06:24.791344 16703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:24.791495 16703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:24.791675 16703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:24.794008 16703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9664a5f408874fe887c0e5cb33888750" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40643 } }
I20260929 02:06:24.794555 16703 leader_election.cc:302] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9664a5f408874fe887c0e5cb33888750; no voters:
I20260929 02:06:24.795300 16703 leader_election.cc:288] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:24.795449 16708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:24.796197 16708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [term 1 LEADER]: Becoming Leader. State: Replica: 9664a5f408874fe887c0e5cb33888750, State: Running, Role: LEADER
I20260929 02:06:24.796813 16708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9664a5f408874fe887c0e5cb33888750" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40643 } }
I20260929 02:06:24.797571 16703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:24.800028 16710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9664a5f408874fe887c0e5cb33888750. Latest consensus state: current_term: 1 leader_uuid: "9664a5f408874fe887c0e5cb33888750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9664a5f408874fe887c0e5cb33888750" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40643 } } }
I20260929 02:06:24.800024 16709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9664a5f408874fe887c0e5cb33888750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9664a5f408874fe887c0e5cb33888750" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40643 } } }
I20260929 02:06:24.800637 16709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:24.800637 16710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:24.803092 16715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:24.809504 16715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:24.819337 16715 catalog_manager.cc:1347] Generated new cluster ID: cc9cf9e21bef4ac6a6197f60992913fe
I20260929 02:06:24.819473 16715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:24.836882 16715 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:24.837366 16715 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:24.843417 16715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750: Generated new TSK 0
I20260929 02:06:24.844410 16715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:06:24.860526 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40643
--builtin_ntp_servers=127.11.182.212:44641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:06:25.246302 16727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:25.246609 16727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:25.246706 16727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:25.255163 16727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:25.255290 16727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:25.255404 16727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:25.265982 16727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:25.267310 16727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:25.269099 16727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:25.285017 16732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:25.284933 16735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:25.284961 16733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:25.286185 16727 server_base.cc:1034] running on GCE node
I20260929 02:06:25.286800 16727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:25.287923 16727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:25.289150 16727 hybrid_clock.cc:648] HybridClock initialized: now 1790647585289100 us; error 38 us; skew 500 ppm
I20260929 02:06:25.289546 16727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:06:25.292426 16727 webserver.cc:460] Webserver started at http://127.11.182.193:34609/ using document root <none> and password file <none>
I20260929 02:06:25.293291 16727 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:25.293396 16727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:25.293756 16727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:25.296785 16727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "fd54caa957804566a528264d29a6f805"
format_stamp: "Formatted at 2026-09-29 02:06:25 on dist-test-slave-q06p"
server_key: "d805bf157c37964a0709bc09666c365a"
server_key_iv: "5fadc33c9a318435696b2d89decb7f9c"
server_key_version: "encryptionkey@0"
I20260929 02:06:25.297688 16727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "fd54caa957804566a528264d29a6f805"
format_stamp: "Formatted at 2026-09-29 02:06:25 on dist-test-slave-q06p"
server_key: "d805bf157c37964a0709bc09666c365a"
server_key_iv: "5fadc33c9a318435696b2d89decb7f9c"
server_key_version: "encryptionkey@0"
I20260929 02:06:25.305325 16727 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 02:06:25.309510 16741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:25.311674 16727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:25.311928 16727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "fd54caa957804566a528264d29a6f805"
format_stamp: "Formatted at 2026-09-29 02:06:25 on dist-test-slave-q06p"
server_key: "d805bf157c37964a0709bc09666c365a"
server_key_iv: "5fadc33c9a318435696b2d89decb7f9c"
server_key_version: "encryptionkey@0"
I20260929 02:06:25.312199 16727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:25.319872 16727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:25.320878 16727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:25.321209 16727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:25.322522 16727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:25.324376 16727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:25.324513 16727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:25.324666 16727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:25.324755 16727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:25.363567 16727 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:41069
I20260929 02:06:25.363629 16853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:41069 every 8 connection(s)
I20260929 02:06:25.365036 16727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:25.367139 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 16727
I20260929 02:06:25.367338 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:25.368115 11995 external_mini_cluster.cc:1428] Setting key f22f953f561dbc602d2396234c461c70
I20260929 02:06:25.372295 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40643
--builtin_ntp_servers=127.11.182.212:44641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 02:06:25.378611 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:25.368727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:41619 (local address 127.11.182.254:40643)
0929 02:06:25.369226 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:25.369232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:25.371541 (+ 2309us) server_negotiation.cc:407] Connection header received
0929 02:06:25.374997 (+ 3456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:25.375003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:25.375062 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:25.375166 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:25.377268 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:25.377274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:25.377277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:25.377301 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:25.377355 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:25.377371 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:25.377527 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:25.378277 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 02:06:25.378353 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":262,"threads_started":1}
I20260929 02:06:25.379797 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:25.369552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40643 (local address 127.11.182.193:41619)
0929 02:06:25.371339 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:25.371400 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:25.374481 (+ 3081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:25.375501 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:25.375517 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:25.375707 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:25.376701 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:06:25.376707 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:25.376789 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:06:25.376936 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:25.377514 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:25.377530 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:25.377553 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:06:25.377911 (+ 358us) client_negotiation.cc:240] Negotiation successful
0929 02:06:25.378262 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":542,"threads_started":1}
I20260929 02:06:25.381718 16854 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40643
I20260929 02:06:25.382093 16854 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:25.382766 16854 heartbeater.cc:511] Master 127.11.182.254:40643 requested a full tablet report, sending...
I20260929 02:06:25.386005 16668 ts_manager.cc:194] Registered new tserver with Master: fd54caa957804566a528264d29a6f805 (127.11.182.193:41069)
W20260929 02:06:25.732986 16858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:25.733368 16858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:25.733469 16858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:25.740953 16858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:25.741080 16858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:25.741200 16858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:25.750728 16858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:25.752032 16858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:25.753748 16858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:25.771255 16864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:25.771863 16866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:25.771312 16863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:25.773689 16858 server_base.cc:1034] running on GCE node
I20260929 02:06:25.774462 16858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:25.775755 16858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:25.776990 16858 hybrid_clock.cc:648] HybridClock initialized: now 1790647585776939 us; error 38 us; skew 500 ppm
I20260929 02:06:25.777366 16858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:06:25.779840 16858 webserver.cc:460] Webserver started at http://127.11.182.194:38149/ using document root <none> and password file <none>
I20260929 02:06:25.780560 16858 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:25.780689 16858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:25.781040 16858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:25.784034 16858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "99d8f18e533548a98a56bf2b9b40a5cf"
format_stamp: "Formatted at 2026-09-29 02:06:25 on dist-test-slave-q06p"
server_key: "b07e15d153100ed0def008eadc319a8f"
server_key_iv: "854335c7448c5eff01284e033de5a3c7"
server_key_version: "encryptionkey@0"
I20260929 02:06:25.784845 16858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "99d8f18e533548a98a56bf2b9b40a5cf"
format_stamp: "Formatted at 2026-09-29 02:06:25 on dist-test-slave-q06p"
server_key: "b07e15d153100ed0def008eadc319a8f"
server_key_iv: "854335c7448c5eff01284e033de5a3c7"
server_key_version: "encryptionkey@0"
I20260929 02:06:25.790467 16858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:06:25.794222 16872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:25.795888 16858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:25.796191 16858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "99d8f18e533548a98a56bf2b9b40a5cf"
format_stamp: "Formatted at 2026-09-29 02:06:25 on dist-test-slave-q06p"
server_key: "b07e15d153100ed0def008eadc319a8f"
server_key_iv: "854335c7448c5eff01284e033de5a3c7"
server_key_version: "encryptionkey@0"
I20260929 02:06:25.796449 16858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:25.803164 16858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:25.804087 16858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:25.804405 16858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:25.805441 16858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:25.807040 16858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:25.807181 16858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:25.807310 16858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:25.807395 16858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:25.842779 16858 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:37271
I20260929 02:06:25.842832 16984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:37271 every 8 connection(s)
I20260929 02:06:25.844149 16858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:25.849193 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 16858
I20260929 02:06:25.849416 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:25.849911 11995 external_mini_cluster.cc:1428] Setting key 9a543ffb793a24faf4da22c0f61bb0a5
I20260929 02:06:25.853283 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40643
--builtin_ntp_servers=127.11.182.212:44641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 02:06:25.853332 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:25.846992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:51035 (local address 127.11.182.254:40643)
0929 02:06:25.847203 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:25.847210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:25.848854 (+ 1644us) server_negotiation.cc:407] Connection header received
0929 02:06:25.850808 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:25.850813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:25.850943 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:25.851069 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:25.852689 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:25.852693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:25.852696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:25.852712 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:25.852743 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:25.852754 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:25.852877 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:25.853087 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 02:06:25.853135 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:06:25.854825 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:25.847616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40643 (local address 127.11.182.194:51035)
0929 02:06:25.848634 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:25.848677 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:25.850481 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:25.851308 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:25.851317 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:25.851427 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:25.852307 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:06:25.852311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:25.852375 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:06:25.852494 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:25.852856 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:25.852867 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:25.852877 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:06:25.853192 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 02:06:25.853496 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":347,"threads_started":1}
I20260929 02:06:25.856359 16985 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40643
I20260929 02:06:25.856691 16985 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:25.857417 16985 heartbeater.cc:511] Master 127.11.182.254:40643 requested a full tablet report, sending...
I20260929 02:06:25.859266 16668 ts_manager.cc:194] Registered new tserver with Master: 99d8f18e533548a98a56bf2b9b40a5cf (127.11.182.194:37271)
W20260929 02:06:26.188694 16988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:26.188993 16988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:26.189081 16988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:26.196566 16988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:26.196681 16988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:26.196799 16988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:26.206449 16988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:26.207890 16988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:26.209656 16988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:26.223584 16993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:26.223647 16994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:26.223628 16996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:26.224709 16988 server_base.cc:1034] running on GCE node
I20260929 02:06:26.225420 16988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:26.226418 16988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:26.227712 16988 hybrid_clock.cc:648] HybridClock initialized: now 1790647586227649 us; error 45 us; skew 500 ppm
I20260929 02:06:26.228116 16988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:06:26.230760 16988 webserver.cc:460] Webserver started at http://127.11.182.195:33727/ using document root <none> and password file <none>
I20260929 02:06:26.231542 16988 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:26.231693 16988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:26.232066 16988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:26.235219 16988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "60409c47892d48588bfc8d318ff2863f"
format_stamp: "Formatted at 2026-09-29 02:06:26 on dist-test-slave-q06p"
server_key: "d540ea9679b202d38d83b2d44effb086"
server_key_iv: "321fd7099bf6a51151c4efc9ea0ad0fd"
server_key_version: "encryptionkey@0"
I20260929 02:06:26.236155 16988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "60409c47892d48588bfc8d318ff2863f"
format_stamp: "Formatted at 2026-09-29 02:06:26 on dist-test-slave-q06p"
server_key: "d540ea9679b202d38d83b2d44effb086"
server_key_iv: "321fd7099bf6a51151c4efc9ea0ad0fd"
server_key_version: "encryptionkey@0"
I20260929 02:06:26.241962 16988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:06:26.245553 17002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:26.247027 16988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:26.247203 16988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "60409c47892d48588bfc8d318ff2863f"
format_stamp: "Formatted at 2026-09-29 02:06:26 on dist-test-slave-q06p"
server_key: "d540ea9679b202d38d83b2d44effb086"
server_key_iv: "321fd7099bf6a51151c4efc9ea0ad0fd"
server_key_version: "encryptionkey@0"
I20260929 02:06:26.247432 16988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:26.253139 16988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:26.253908 16988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:26.254185 16988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:26.255230 16988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:26.256660 16988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:26.256772 16988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:26.256901 16988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:26.256991 16988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:26.290105 16988 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:37385
I20260929 02:06:26.290165 17114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:37385 every 8 connection(s)
I20260929 02:06:26.291540 16988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:26.295773 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 16988
I20260929 02:06:26.295971 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:26.296520 11995 external_mini_cluster.cc:1428] Setting key ff6ac0bc539828f9a7a998fe64d59aac
I20260929 02:06:26.300760 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.294150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:33927 (local address 127.11.182.254:40643)
0929 02:06:26.294363 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:26.294370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:26.296058 (+ 1688us) server_negotiation.cc:407] Connection header received
0929 02:06:26.298385 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:26.298391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:26.298466 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:26.298563 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:26.300005 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:26.300009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:26.300012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:26.300027 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:26.300061 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:26.300072 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:26.300175 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:26.300407 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 02:06:26.300470 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:06:26.301720 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.294748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40643 (local address 127.11.182.195:33927)
0929 02:06:26.295865 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:26.295921 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:26.298005 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:26.298820 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:26.298827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:26.298986 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:26.299714 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:06:26.299717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:26.299753 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:06:26.299854 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:26.300181 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:26.300190 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:26.300199 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:06:26.300432 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 02:06:26.300712 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":268,"threads_started":1}
I20260929 02:06:26.302997 17115 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40643
I20260929 02:06:26.303282 17115 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:26.303843 17115 heartbeater.cc:511] Master 127.11.182.254:40643 requested a full tablet report, sending...
I20260929 02:06:26.305536 16668 ts_manager.cc:194] Registered new tserver with Master: 60409c47892d48588bfc8d318ff2863f (127.11.182.195:37385)
I20260929 02:06:26.311995 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:06:26.319176 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.316389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.11.182.254:40643)
0929 02:06:26.316548 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:26.316552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:26.317144 (+ 592us) server_negotiation.cc:407] Connection header received
0929 02:06:26.317267 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:26.317271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:26.317323 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:26.317407 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:26.318502 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:26.318506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:26.318510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:26.318524 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:26.318555 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:26.318572 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:26.318690 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:26.318760 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 02:06:26.318806 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:06:26.323251 16668 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:26.330029 16668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:26.364960 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.343213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.11.182.193:41069)
0929 02:06:26.360608 (+ 17395us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:26.360617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:26.360648 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:06:26.360717 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:26.360725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:26.360997 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:26.361159 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:26.363855 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:26.363864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:26.363882 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:26.363903 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:26.363977 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:26.364002 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:26.364153 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:26.364426 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 02:06:26.364538 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17208,"thread_start_us":17608,"threads_started":1}
I20260929 02:06:26.365001 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.342916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:41069 (local address 127.0.0.1:32890)
0929 02:06:26.359965 (+ 17049us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:26.359984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:26.360101 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:26.362296 (+ 2195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:26.362303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:26.362404 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:26.363559 (+ 1155us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:06:26.363562 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:26.363592 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:06:26.363680 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:26.364225 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:26.364239 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:26.364248 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:06:26.364421 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 02:06:26.364529 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16901,"thread_start_us":15613,"threads_started":1}
I20260929 02:06:26.366830 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.342888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.11.182.195:37385)
0929 02:06:26.362538 (+ 19650us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:26.362545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:26.362579 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:06:26.362652 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:26.362661 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:26.362889 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:26.363041 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:26.365773 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:26.365785 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:26.365800 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:26.365821 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:26.365886 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:26.365909 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:26.366093 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:26.366327 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 02:06:26.366438 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19484,"thread_start_us":19324,"threads_started":1}
I20260929 02:06:26.367007 17127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.342657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:37385 (local address 127.0.0.1:46270)
0929 02:06:26.359155 (+ 16498us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:26.359231 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:26.359459 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:26.364657 (+ 5198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:26.364662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:26.364696 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:26.365475 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:06:26.365481 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:26.365500 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:06:26.365558 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:26.366153 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:26.366159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:26.366166 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:06:26.366326 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 02:06:26.366516 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16059,"thread_start_us":16992,"threads_started":1}
I20260929 02:06:26.368535 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.343289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:37271 (local address 127.0.0.1:56412)
0929 02:06:26.361864 (+ 18575us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:26.361881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:26.361986 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:26.365751 (+ 3765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:26.365760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:26.365785 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:26.367263 (+ 1478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:06:26.367267 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:26.367282 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:06:26.367416 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:26.367923 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:26.367928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:26.367932 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 02:06:26.368065 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 02:06:26.368115 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18462,"thread_start_us":18307,"threads_started":1}
I20260929 02:06:26.369232 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.343541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.11.182.194:37271)
0929 02:06:26.361464 (+ 17923us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:26.361472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:26.362006 (+ 534us) server_negotiation.cc:407] Connection header received
0929 02:06:26.362110 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:26.362131 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:26.362375 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:26.362934 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:26.367602 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:26.367611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:26.367625 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:26.367647 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:26.367717 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:26.367742 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:26.367873 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:26.368797 (+ 924us) server_negotiation.cc:299] Negotiation successful
0929 02:06:26.368908 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17771,"thread_start_us":17658,"threads_started":1}
I20260929 02:06:26.369647 16789 tablet_service.cc:1467] Processing CreateTablet for tablet ca26e2d19faf465f841ca5a32dc113ac (DEFAULT_TABLE table=test-table [id=5a642d3a28be4c469db50271f3152ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:26.371337 16789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca26e2d19faf465f841ca5a32dc113ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:26.372807 17050 tablet_service.cc:1467] Processing CreateTablet for tablet ca26e2d19faf465f841ca5a32dc113ac (DEFAULT_TABLE table=test-table [id=5a642d3a28be4c469db50271f3152ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:26.372895 16920 tablet_service.cc:1467] Processing CreateTablet for tablet ca26e2d19faf465f841ca5a32dc113ac (DEFAULT_TABLE table=test-table [id=5a642d3a28be4c469db50271f3152ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:26.374155 16920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca26e2d19faf465f841ca5a32dc113ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:26.374547 17050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca26e2d19faf465f841ca5a32dc113ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:26.386075 17133 tablet_bootstrap.cc:492] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: Bootstrap starting.
I20260929 02:06:26.387111 17134 tablet_bootstrap.cc:492] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: Bootstrap starting.
I20260929 02:06:26.389055 17135 tablet_bootstrap.cc:492] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: Bootstrap starting.
I20260929 02:06:26.389940 16854 heartbeater.cc:503] Master 127.11.182.254:40643 was elected leader, sending a full tablet report...
I20260929 02:06:26.391008 17134 tablet_bootstrap.cc:654] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:26.391004 17133 tablet_bootstrap.cc:654] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:26.392328 17134 log.cc:824] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:26.392323 17133 log.cc:824] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:26.396538 17134 tablet_bootstrap.cc:492] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: No bootstrap required, opened a new log
I20260929 02:06:26.396857 17134 ts_tablet_manager.cc:1397] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 02:06:26.397646 17135 tablet_bootstrap.cc:654] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:26.399432 17135 log.cc:824] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:26.403302 17133 tablet_bootstrap.cc:492] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: No bootstrap required, opened a new log
I20260929 02:06:26.403713 17133 ts_tablet_manager.cc:1397] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20260929 02:06:26.405524 17134 raft_consensus.cc:348] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } }
I20260929 02:06:26.405934 17134 raft_consensus.cc:374] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:26.406073 17134 raft_consensus.cc:729] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd54caa957804566a528264d29a6f805, State: Initialized, Role: FOLLOWER
I20260929 02:06:26.406770 17134 consensus_queue.cc:260] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } }
I20260929 02:06:26.409456 17135 tablet_bootstrap.cc:492] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: No bootstrap required, opened a new log
I20260929 02:06:26.409489 17134 ts_tablet_manager.cc:1428] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260929 02:06:26.410037 17135 ts_tablet_manager.cc:1397] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.005s
I20260929 02:06:26.415489 17133 raft_consensus.cc:348] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } }
I20260929 02:06:26.416183 17133 raft_consensus.cc:374] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:26.416335 17133 raft_consensus.cc:729] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d8f18e533548a98a56bf2b9b40a5cf, State: Initialized, Role: FOLLOWER
I20260929 02:06:26.417129 17133 consensus_queue.cc:260] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } }
I20260929 02:06:26.418380 16985 heartbeater.cc:503] Master 127.11.182.254:40643 was elected leader, sending a full tablet report...
I20260929 02:06:26.419104 17133 ts_tablet_manager.cc:1428] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260929 02:06:26.421007 17135 raft_consensus.cc:348] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } }
I20260929 02:06:26.421419 17135 raft_consensus.cc:374] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:26.421530 17135 raft_consensus.cc:729] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60409c47892d48588bfc8d318ff2863f, State: Initialized, Role: FOLLOWER
I20260929 02:06:26.422151 17135 consensus_queue.cc:260] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } }
I20260929 02:06:26.423256 17115 heartbeater.cc:503] Master 127.11.182.254:40643 was elected leader, sending a full tablet report...
I20260929 02:06:26.423916 17135 ts_tablet_manager.cc:1428] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260929 02:06:26.543311 17116 tablet.cc:2369] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:06:26.596369 16986 tablet.cc:2369] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:06:26.617862 16855 tablet.cc:2369] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:06:26.751194 17139 raft_consensus.cc:482] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:26.751571 17139 raft_consensus.cc:504] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } }
I20260929 02:06:26.753345 17139 leader_election.cc:288] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60409c47892d48588bfc8d318ff2863f (127.11.182.195:37385), 99d8f18e533548a98a56bf2b9b40a5cf (127.11.182.194:37271)
I20260929 02:06:26.756826 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.753765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:42415 (local address 127.11.182.195:37385)
0929 02:06:26.754365 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:26.754374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:26.754626 (+ 252us) server_negotiation.cc:407] Connection header received
0929 02:06:26.754902 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:26.754907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:26.754976 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:26.755094 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:26.756203 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:26.756209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:26.756214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:26.756233 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:26.756273 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:26.756288 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:26.756411 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:26.756510 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 02:06:26.756565 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433}
I20260929 02:06:26.757064 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.753590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:37385 (local address 127.11.182.193:42415)
0929 02:06:26.754477 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:26.754502 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:26.754679 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:26.755136 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:26.755144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:26.755169 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:26.755983 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:06:26.755985 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:26.755999 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:06:26.756065 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:26.756407 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:26.756412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:26.756416 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 02:06:26.756556 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 02:06:26.756609 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":503,"threads_started":1}
I20260929 02:06:26.757138 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.753913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:54989 (local address 127.11.182.194:37271)
0929 02:06:26.754155 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:26.754165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:26.754991 (+ 826us) server_negotiation.cc:407] Connection header received
0929 02:06:26.755188 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:26.755194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:26.755257 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:26.755350 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:26.756400 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:26.756405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:26.756409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:26.756430 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:26.756470 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:26.756494 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:26.756600 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:26.756794 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 02:06:26.756865 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 02:06:26.757863 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.753791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:37271 (local address 127.11.182.193:54989)
0929 02:06:26.754865 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:26.754897 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:26.755023 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:26.755566 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:26.755570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:26.755597 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:26.756206 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:06:26.756209 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:26.756230 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:06:26.756258 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:26.756648 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:26.756654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:26.756661 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:06:26.757378 (+ 717us) client_negotiation.cc:240] Negotiation successful
0929 02:06:26.757437 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":690,"threads_started":1}
I20260929 02:06:26.758193 17070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca26e2d19faf465f841ca5a32dc113ac" candidate_uuid: "fd54caa957804566a528264d29a6f805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60409c47892d48588bfc8d318ff2863f" is_pre_election: true
I20260929 02:06:26.758718 17070 raft_consensus.cc:2454] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd54caa957804566a528264d29a6f805 in term 0.
I20260929 02:06:26.759479 16743 leader_election.cc:302] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60409c47892d48588bfc8d318ff2863f, fd54caa957804566a528264d29a6f805; no voters:
I20260929 02:06:26.759979 17139 raft_consensus.cc:2781] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:26.760128 17139 raft_consensus.cc:482] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:26.760262 17139 raft_consensus.cc:3037] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:26.760064 16940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca26e2d19faf465f841ca5a32dc113ac" candidate_uuid: "fd54caa957804566a528264d29a6f805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" is_pre_election: true
I20260929 02:06:26.760612 16940 raft_consensus.cc:2454] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd54caa957804566a528264d29a6f805 in term 0.
I20260929 02:06:26.762621 17139 raft_consensus.cc:504] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } }
I20260929 02:06:26.763283 17139 leader_election.cc:288] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [CANDIDATE]: Term 1 election: Requested vote from peers 60409c47892d48588bfc8d318ff2863f (127.11.182.195:37385), 99d8f18e533548a98a56bf2b9b40a5cf (127.11.182.194:37271)
I20260929 02:06:26.763726 17070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca26e2d19faf465f841ca5a32dc113ac" candidate_uuid: "fd54caa957804566a528264d29a6f805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60409c47892d48588bfc8d318ff2863f"
I20260929 02:06:26.763748 16940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca26e2d19faf465f841ca5a32dc113ac" candidate_uuid: "fd54caa957804566a528264d29a6f805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d8f18e533548a98a56bf2b9b40a5cf"
I20260929 02:06:26.763952 16940 raft_consensus.cc:3037] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:26.763965 17070 raft_consensus.cc:3037] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:26.766112 17070 raft_consensus.cc:2454] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd54caa957804566a528264d29a6f805 in term 1.
I20260929 02:06:26.766516 16743 leader_election.cc:302] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60409c47892d48588bfc8d318ff2863f, fd54caa957804566a528264d29a6f805; no voters:
I20260929 02:06:26.766815 17139 raft_consensus.cc:2781] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:26.767133 16940 raft_consensus.cc:2454] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd54caa957804566a528264d29a6f805 in term 1.
I20260929 02:06:26.767622 17139 raft_consensus.cc:686] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 1 LEADER]: Becoming Leader. State: Replica: fd54caa957804566a528264d29a6f805, State: Running, Role: LEADER
I20260929 02:06:26.768260 17139 consensus_queue.cc:237] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } }
I20260929 02:06:26.773273 16667 catalog_manager.cc:5495] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd54caa957804566a528264d29a6f805 (127.11.182.193). New cstate: current_term: 1 leader_uuid: "fd54caa957804566a528264d29a6f805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd54caa957804566a528264d29a6f805" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 } health_report { overall_health: UNKNOWN } } }
I20260929 02:06:26.827236 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.823911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.11.182.193:41069)
0929 02:06:26.824137 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:26.824147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:26.824597 (+ 450us) server_negotiation.cc:407] Connection header received
0929 02:06:26.824786 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:26.824790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:26.824893 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:26.824979 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:26.826209 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:26.826214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:26.826217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:26.826234 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:26.826277 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:26.826289 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:26.826399 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:26.826651 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 02:06:26.826748 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:06:26.836025 16940 raft_consensus.cc:1261] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [term 1 FOLLOWER]: Refusing update from remote peer fd54caa957804566a528264d29a6f805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:26.836025 17070 raft_consensus.cc:1261] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [term 1 FOLLOWER]: Refusing update from remote peer fd54caa957804566a528264d29a6f805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:26.837157 17144 consensus_queue.cc:1035] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60409c47892d48588bfc8d318ff2863f" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 37385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:26.837555 17139 consensus_queue.cc:1035] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d8f18e533548a98a56bf2b9b40a5cf" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:26.877818 16667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:26.878093 16667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:26.881883 16667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9664a5f408874fe887c0e5cb33888750: Sending DeleteTablet for 3 replicas of tablet ca26e2d19faf465f841ca5a32dc113ac
I20260929 02:06:26.883287 17050 tablet_service.cc:1514] Processing DeleteTablet for tablet ca26e2d19faf465f841ca5a32dc113ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:26 UTC) from {username='slave'} at 127.0.0.1:46270
I20260929 02:06:26.883284 16920 tablet_service.cc:1514] Processing DeleteTablet for tablet ca26e2d19faf465f841ca5a32dc113ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:26 UTC) from {username='slave'} at 127.0.0.1:56412
I20260929 02:06:26.883423 16789 tablet_service.cc:1514] Processing DeleteTablet for tablet ca26e2d19faf465f841ca5a32dc113ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:26 UTC) from {username='slave'} at 127.0.0.1:32890
I20260929 02:06:26.888043 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 16727
I20260929 02:06:26.898856 17160 tablet_replica.cc:330] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: stopping tablet replica
I20260929 02:06:26.899622 17160 raft_consensus.cc:2229] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:26.900759 17160 raft_consensus.cc:2258] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:26.903445 17160 ts_tablet_manager.cc:1905] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:26.912045 17160 ts_tablet_manager.cc:1918] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:26.912380 17160 log.cc:1196] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/ca26e2d19faf465f841ca5a32dc113ac
I20260929 02:06:26.913070 17160 ts_tablet_manager.cc:1939] T ca26e2d19faf465f841ca5a32dc113ac P fd54caa957804566a528264d29a6f805: Deleting consensus metadata
I20260929 02:06:26.915365 16655 catalog_manager.cc:4841] TS fd54caa957804566a528264d29a6f805 (127.11.182.193:41069): tablet ca26e2d19faf465f841ca5a32dc113ac (table test-table [id=5a642d3a28be4c469db50271f3152ecc]) successfully deleted
I20260929 02:06:26.916466 17159 tablet_replica.cc:330] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: stopping tablet replica
I20260929 02:06:26.917522 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:26.891124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.11.182.193:41069)
0929 02:06:26.891431 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:26.891441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:26.914058 (+ 22617us) server_negotiation.cc:407] Connection header received
0929 02:06:26.914796 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:26.914806 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:26.914997 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:26.915097 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:26.916616 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:26.916622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:26.916627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:26.916650 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:26.916690 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:26.916719 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:26.916831 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:26.917127 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 02:06:26.917206 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:06:26.917248 17159 raft_consensus.cc:2229] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:26.918887 16849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:26.918908 17159 raft_consensus.cc:2258] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:26.921921 17159 ts_tablet_manager.cc:1905] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:26.923553 17158 tablet_replica.cc:330] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: stopping tablet replica
I20260929 02:06:26.924024 17158 raft_consensus.cc:2229] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:26.924779 17158 raft_consensus.cc:2258] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:26.926993 17158 ts_tablet_manager.cc:1905] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:26.930408 17159 ts_tablet_manager.cc:1918] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:26.930594 17159 log.cc:1196] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/ca26e2d19faf465f841ca5a32dc113ac
I20260929 02:06:26.931221 17159 ts_tablet_manager.cc:1939] T ca26e2d19faf465f841ca5a32dc113ac P 60409c47892d48588bfc8d318ff2863f: Deleting consensus metadata
I20260929 02:06:26.933359 16653 catalog_manager.cc:4841] TS 60409c47892d48588bfc8d318ff2863f (127.11.182.195:37385): tablet ca26e2d19faf465f841ca5a32dc113ac (table test-table [id=5a642d3a28be4c469db50271f3152ecc]) successfully deleted
I20260929 02:06:26.934193 17158 ts_tablet_manager.cc:1918] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:26.934384 17158 log.cc:1196] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/ca26e2d19faf465f841ca5a32dc113ac
I20260929 02:06:26.934890 17158 ts_tablet_manager.cc:1939] T ca26e2d19faf465f841ca5a32dc113ac P 99d8f18e533548a98a56bf2b9b40a5cf: Deleting consensus metadata
I20260929 02:06:26.936587 16652 catalog_manager.cc:4841] TS 99d8f18e533548a98a56bf2b9b40a5cf (127.11.182.194:37271): tablet ca26e2d19faf465f841ca5a32dc113ac (table test-table [id=5a642d3a28be4c469db50271f3152ecc]) successfully deleted
I20260929 02:06:27.018939 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 16727
I20260929 02:06:27.038368 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 16858
I20260929 02:06:27.041397 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:27.039264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.11.182.194:37271)
0929 02:06:27.039454 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:27.039458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:27.039481 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:06:27.039629 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:27.039633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:27.039719 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:27.039803 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:27.040604 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:27.040608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:27.040611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:27.040625 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:27.040654 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:27.040667 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:27.040766 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:27.040880 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 02:06:27.040963 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:06:27.042071 16980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:27.134999 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 16858
I20260929 02:06:27.152371 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 16988
I20260929 02:06:27.154928 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:27.153154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.11.182.195:37385)
0929 02:06:27.153316 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:27.153322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:27.153341 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:06:27.153415 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:27.153418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:27.153504 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:27.153585 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:27.154363 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:27.154367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:27.154371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:27.154385 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:27.154415 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:27.154426 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:27.154524 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:27.154596 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 02:06:27.154639 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:06:27.155596 17110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:27.245198 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 16988
I20260929 02:06:27.262430 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 16637
I20260929 02:06:27.263684 16698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:27.372306 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 16637
2026-09-29T02:06:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3106 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 02:06:27.414438 11995 mini_oidc.cc:152] Starting JWKS server
I20260929 02:06:27.417366 11995 webserver.cc:460] Webserver started at https://127.0.0.1:40465/ using document root <none> and password file <none>
I20260929 02:06:27.417691 11995 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:06:27.418367 11995 webserver.cc:460] Webserver started at http://127.0.0.1:41803/ using document root <none> and password file <none>
2026-09-29T02:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:27Z Disabled control of system clock
I20260929 02:06:27.447274 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41653
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:41653
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40465/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/testchainca.cert with env {}
W20260929 02:06:27.799963 17171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:27.800457 17171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:27.800648 17171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:27.808933 17171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:27.809065 17171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:27.809103 17171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:27.809156 17171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:27.809201 17171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:06:27.809263 17171 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:06:27.809302 17171 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40465/jwks.json
I20260929 02:06:27.820590 17171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:41653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40465/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:27.822109 17171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:27.823606 17171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:27.836143 17176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:27.836035 17179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:27.836150 17177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:27.837167 17171 server_base.cc:1034] running on GCE node
I20260929 02:06:27.838081 17171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:27.839701 17171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:27.840970 17171 hybrid_clock.cc:648] HybridClock initialized: now 1790647587840914 us; error 35 us; skew 500 ppm
I20260929 02:06:27.841322 17171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:06:27.848650 17171 webserver.cc:460] Webserver started at http://127.11.182.254:33793/ using document root <none> and password file <none>
I20260929 02:06:27.849417 17171 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:27.849514 17171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:27.849838 17171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:27.853833 17171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "b2cf927886c64fc4b2dc417ae378b261"
format_stamp: "Formatted at 2026-09-29 02:06:27 on dist-test-slave-q06p"
server_key: "720145365d01fb40b5a18a6d2a6c3ec9"
server_key_iv: "4b4f6603146420525044b1265ca5c280"
server_key_version: "encryptionkey@0"
I20260929 02:06:27.854620 17171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "b2cf927886c64fc4b2dc417ae378b261"
format_stamp: "Formatted at 2026-09-29 02:06:27 on dist-test-slave-q06p"
server_key: "720145365d01fb40b5a18a6d2a6c3ec9"
server_key_iv: "4b4f6603146420525044b1265ca5c280"
server_key_version: "encryptionkey@0"
I20260929 02:06:27.860962 17171 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260929 02:06:27.864766 17185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:27.866297 17171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:27.866459 17171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "b2cf927886c64fc4b2dc417ae378b261"
format_stamp: "Formatted at 2026-09-29 02:06:27 on dist-test-slave-q06p"
server_key: "720145365d01fb40b5a18a6d2a6c3ec9"
server_key_iv: "4b4f6603146420525044b1265ca5c280"
server_key_version: "encryptionkey@0"
I20260929 02:06:27.866648 17171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:27.911369 17171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:27.912199 17171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:27.912477 17171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:27.931439 17171 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:41653
I20260929 02:06:27.931478 17236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:41653 every 8 connection(s)
I20260929 02:06:27.932853 17171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:27.937028 17237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:27.939904 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 17171
I20260929 02:06:27.940150 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:27.940948 11995 external_mini_cluster.cc:1428] Setting key 582b6f1c772bd16a9f8ba047004614e3
I20260929 02:06:27.950176 17237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261: Bootstrap starting.
I20260929 02:06:27.953960 17237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:27.954998 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:27.943093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.11.182.254:41653)
0929 02:06:27.944261 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:27.944272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:27.944317 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:06:27.945967 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:27.945990 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:27.946467 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:27.946978 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:27.947580 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:27.948935 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:27.950255 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:27.951104 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:27.951310 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:27.952600 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:27.952607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:27.952621 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:27.953158 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:27.953233 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:27.953253 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:27.953384 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:27.953610 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 02:06:27.953933 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":419,"threads_started":1}
I20260929 02:06:27.955278 17237 log.cc:824] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:27.958635 17237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261: No bootstrap required, opened a new log
I20260929 02:06:27.959141 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:06:27.967103 17237 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cf927886c64fc4b2dc417ae378b261" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41653 } }
I20260929 02:06:27.967511 17237 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:27.967634 17237 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cf927886c64fc4b2dc417ae378b261, State: Initialized, Role: FOLLOWER
I20260929 02:06:27.968273 17237 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cf927886c64fc4b2dc417ae378b261" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41653 } }
I20260929 02:06:27.968526 17237 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:27.968675 17237 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:27.968868 17237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:27.971788 17237 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cf927886c64fc4b2dc417ae378b261" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41653 } }
I20260929 02:06:27.972357 17237 leader_election.cc:302] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2cf927886c64fc4b2dc417ae378b261; no voters:
I20260929 02:06:27.973089 17237 leader_election.cc:288] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:27.973212 17242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:27.973950 17242 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [term 1 LEADER]: Becoming Leader. State: Replica: b2cf927886c64fc4b2dc417ae378b261, State: Running, Role: LEADER
I20260929 02:06:27.974536 17242 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cf927886c64fc4b2dc417ae378b261" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41653 } }
I20260929 02:06:27.975517 17237 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:27.977962 17243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2cf927886c64fc4b2dc417ae378b261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cf927886c64fc4b2dc417ae378b261" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41653 } } }
I20260929 02:06:27.978080 17244 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2cf927886c64fc4b2dc417ae378b261. Latest consensus state: current_term: 1 leader_uuid: "b2cf927886c64fc4b2dc417ae378b261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cf927886c64fc4b2dc417ae378b261" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41653 } } }
I20260929 02:06:27.978521 17244 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:27.978386 17243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:27.984058 17247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:27.989275 17247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:27.997861 17247 catalog_manager.cc:1347] Generated new cluster ID: 7b725cc0e4044b32a1d524c66d0a7371
I20260929 02:06:27.997984 17247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:28.032665 17247 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:28.034507 17247 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:28.043989 17247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2cf927886c64fc4b2dc417ae378b261: Generated new TSK 0
I20260929 02:06:28.044898 17247 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:06:28.095391 17267 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 02:06:28.095577 17267 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:06:28.096555 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:28.067434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.11.182.254:41653)
0929 02:06:28.067663 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:28.067670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:28.068125 (+ 455us) server_negotiation.cc:407] Connection header received
0929 02:06:28.068487 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:28.068492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:28.068567 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:28.068723 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:06:28.069270 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.069985 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.071685 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.072163 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.072308 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:28.073022 (+ 714us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:06:28.095826 (+ 22804us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:06:28.096235 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37822: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 02:06:28.096688 17240 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37822: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 02:06:28.098084 17266 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:28.067349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41653 (local address 127.0.0.1:37822)
0929 02:06:28.068013 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:28.068029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:28.068245 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:28.068767 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:28.068772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:28.068784 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:06:28.069108 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:28.069117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.070136 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.070144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:28.071528 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:28.071539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.072330 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.072335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:28.072501 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:28.072559 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:06:28.096163 (+ 23604us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:06:28.096516 (+ 353us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.182.254:41653: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":367,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.182.254:41653: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 02:06:28.100033 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 17171
I20260929 02:06:28.101284 17232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:28.197078 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 17171
2026-09-29T02:06:28Z chronyd exiting
I20260929 02:06:28.247804 11995 test_util.cc:175] -----------------------------------------------
I20260929 02:06:28.247991 11995 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647540175441-11995-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (838 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 02:06:28.251847 11995 mini_oidc.cc:152] Starting JWKS server
I20260929 02:06:28.253693 11995 webserver.cc:460] Webserver started at https://127.0.0.1:35479/ using document root <none> and password file <none>
I20260929 02:06:28.253938 11995 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:06:28.254628 11995 webserver.cc:460] Webserver started at http://127.0.0.1:38225/ using document root <none> and password file <none>
2026-09-29T02:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:28Z Disabled control of system clock
I20260929 02:06:28.282392 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:44047
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:44047
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35479/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/testchainca.cert with env {}
W20260929 02:06:28.615608 17274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:28.615953 17274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:28.616107 17274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:28.623534 17274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:28.623736 17274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:28.623832 17274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:28.623924 17274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:28.624008 17274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:06:28.624096 17274 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:06:28.624154 17274 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35479/jwks.json
I20260929 02:06:28.634195 17274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:44047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:44047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35479/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:28.635636 17274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:28.637003 17274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:28.648591 17279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:28.648615 17282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:28.648552 17280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:28.649279 17274 server_base.cc:1034] running on GCE node
I20260929 02:06:28.650172 17274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:28.651716 17274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:28.652922 17274 hybrid_clock.cc:648] HybridClock initialized: now 1790647588652873 us; error 55 us; skew 500 ppm
I20260929 02:06:28.653254 17274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:06:28.655638 17274 webserver.cc:460] Webserver started at http://127.11.182.254:39733/ using document root <none> and password file <none>
I20260929 02:06:28.656394 17274 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:28.656519 17274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:28.656838 17274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:28.659829 17274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "e9489ff0ab4b40c59398be44292adc6a"
format_stamp: "Formatted at 2026-09-29 02:06:28 on dist-test-slave-q06p"
server_key: "5a5d5e83b9bc051f104ffa43b8f897be"
server_key_iv: "8dfcec401f8670c604c0232b9905f3f5"
server_key_version: "encryptionkey@0"
I20260929 02:06:28.660662 17274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "e9489ff0ab4b40c59398be44292adc6a"
format_stamp: "Formatted at 2026-09-29 02:06:28 on dist-test-slave-q06p"
server_key: "5a5d5e83b9bc051f104ffa43b8f897be"
server_key_iv: "8dfcec401f8670c604c0232b9905f3f5"
server_key_version: "encryptionkey@0"
I20260929 02:06:28.665935 17274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:06:28.669610 17288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:28.671001 17274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:28.671192 17274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "e9489ff0ab4b40c59398be44292adc6a"
format_stamp: "Formatted at 2026-09-29 02:06:28 on dist-test-slave-q06p"
server_key: "5a5d5e83b9bc051f104ffa43b8f897be"
server_key_iv: "8dfcec401f8670c604c0232b9905f3f5"
server_key_version: "encryptionkey@0"
I20260929 02:06:28.671423 17274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:28.689100 17274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:28.690004 17274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:28.690325 17274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:28.709205 17274 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:44047
I20260929 02:06:28.709225 17339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:44047 every 8 connection(s)
I20260929 02:06:28.710626 17274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:28.712235 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 17274
I20260929 02:06:28.712422 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:28.713056 11995 external_mini_cluster.cc:1428] Setting key 707774a993962f353a65d06992d2bd94
I20260929 02:06:28.714838 17340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:28.726730 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:28.714788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.11.182.254:44047)
0929 02:06:28.715840 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:28.715850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:28.715890 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:06:28.717414 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:28.717437 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:28.717933 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:28.718317 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:28.718980 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.720372 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.722167 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.722823 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.723015 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:28.724222 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:28.724230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:28.724248 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:28.724867 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:28.724958 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:28.724983 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:28.725078 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:28.725330 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 02:06:28.725752 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":375,"threads_started":1}
I20260929 02:06:28.727831 17340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a: Bootstrap starting.
I20260929 02:06:28.730432 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:06:28.731251 17340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:28.732487 17340 log.cc:824] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:28.735666 17340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a: No bootstrap required, opened a new log
I20260929 02:06:28.742758 17340 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9489ff0ab4b40c59398be44292adc6a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44047 } }
I20260929 02:06:28.743193 17340 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:28.743309 17340 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9489ff0ab4b40c59398be44292adc6a, State: Initialized, Role: FOLLOWER
I20260929 02:06:28.743884 17340 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9489ff0ab4b40c59398be44292adc6a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44047 } }
I20260929 02:06:28.744128 17340 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:28.744292 17340 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:28.744544 17340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:28.746731 17340 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9489ff0ab4b40c59398be44292adc6a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44047 } }
I20260929 02:06:28.747291 17340 leader_election.cc:302] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9489ff0ab4b40c59398be44292adc6a; no voters:
I20260929 02:06:28.747910 17340 leader_election.cc:288] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:28.748044 17345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:28.748739 17345 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [term 1 LEADER]: Becoming Leader. State: Replica: e9489ff0ab4b40c59398be44292adc6a, State: Running, Role: LEADER
I20260929 02:06:28.749301 17345 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9489ff0ab4b40c59398be44292adc6a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44047 } }
I20260929 02:06:28.750044 17340 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:28.752233 17347 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9489ff0ab4b40c59398be44292adc6a. Latest consensus state: current_term: 1 leader_uuid: "e9489ff0ab4b40c59398be44292adc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9489ff0ab4b40c59398be44292adc6a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44047 } } }
I20260929 02:06:28.752719 17347 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:28.752362 17346 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9489ff0ab4b40c59398be44292adc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9489ff0ab4b40c59398be44292adc6a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44047 } } }
I20260929 02:06:28.753253 17346 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:28.753794 17353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:28.759481 17353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:28.769308 17353 catalog_manager.cc:1347] Generated new cluster ID: 64bca869ee5a48bbb3f4e8a0faceab99
I20260929 02:06:28.769457 17353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:28.791662 17353 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:28.792999 17353 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:28.807246 17353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9489ff0ab4b40c59398be44292adc6a: Generated new TSK 0
I20260929 02:06:28.808279 17353 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:06:28.848870 17370 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 02:06:28.849010 17370 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:06:28.850346 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:28.822935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.11.182.254:44047)
0929 02:06:28.823127 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:28.823136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:28.823571 (+ 435us) server_negotiation.cc:407] Connection header received
0929 02:06:28.823842 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:28.823846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:28.823935 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:28.824111 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:06:28.824655 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.825446 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.826995 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.827483 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.827679 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:28.828263 (+ 584us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:06:28.849584 (+ 21321us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:06:28.850008 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 02:06:28.850481 17343 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 02:06:28.850539 17369 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:28.822752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44047 (local address 127.0.0.1:33236)
0929 02:06:28.823461 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:28.823479 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:28.823651 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:28.824129 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:28.824135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:28.824149 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:06:28.824455 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:28.824466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.825611 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.825619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:28.826792 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:28.826805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:28.827733 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:28.827741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:28.827924 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:28.827967 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:06:28.849830 (+ 21863us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:06:28.850098 (+ 268us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.182.254:44047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":416,"threads_started":1}
I20260929 02:06:28.852312 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 17274
I20260929 02:06:28.853437 17335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:28.952200 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 17274
2026-09-29T02:06:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (757 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 02:06:29.009233 11995 mini_oidc.cc:152] Starting JWKS server
I20260929 02:06:29.011183 11995 webserver.cc:460] Webserver started at https://127.0.0.1:37017/ using document root <none> and password file <none>
I20260929 02:06:29.011433 11995 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:06:29.012080 11995 webserver.cc:460] Webserver started at http://127.0.0.1:43113/ using document root <none> and password file <none>
2026-09-29T02:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:29Z Disabled control of system clock
I20260929 02:06:29.041887 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:46195
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:46195
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37017/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:06:29.395555 17377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:29.395956 17377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:29.396123 17377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:29.404433 17377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:29.404636 17377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:29.404727 17377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:29.404832 17377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:29.404908 17377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:06:29.405002 17377 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:06:29.405074 17377 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37017/jwks.json
I20260929 02:06:29.415432 17377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:46195
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:46195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37017/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:29.417243 17377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:29.419531 17377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:29.432407 17382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:29.432351 17383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:29.432348 17385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:29.433403 17377 server_base.cc:1034] running on GCE node
I20260929 02:06:29.434264 17377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:29.435998 17377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:29.437310 17377 hybrid_clock.cc:648] HybridClock initialized: now 1790647589437254 us; error 47 us; skew 500 ppm
I20260929 02:06:29.437770 17377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:06:29.440542 17377 webserver.cc:460] Webserver started at http://127.11.182.254:37559/ using document root <none> and password file <none>
I20260929 02:06:29.441305 17377 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:29.441435 17377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:29.441774 17377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:29.445073 17377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "64a1c69c2ae640bd9342fdd944809a01"
format_stamp: "Formatted at 2026-09-29 02:06:29 on dist-test-slave-q06p"
server_key: "859410812bb0c8a12b8bb4e3ecc0f6ac"
server_key_iv: "9e23325e1e9f59559be983c0618ea4b1"
server_key_version: "encryptionkey@0"
I20260929 02:06:29.446066 17377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "64a1c69c2ae640bd9342fdd944809a01"
format_stamp: "Formatted at 2026-09-29 02:06:29 on dist-test-slave-q06p"
server_key: "859410812bb0c8a12b8bb4e3ecc0f6ac"
server_key_iv: "9e23325e1e9f59559be983c0618ea4b1"
server_key_version: "encryptionkey@0"
I20260929 02:06:29.451850 17377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:06:29.455935 17391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:29.457587 17377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:06:29.457808 17377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "64a1c69c2ae640bd9342fdd944809a01"
format_stamp: "Formatted at 2026-09-29 02:06:29 on dist-test-slave-q06p"
server_key: "859410812bb0c8a12b8bb4e3ecc0f6ac"
server_key_iv: "9e23325e1e9f59559be983c0618ea4b1"
server_key_version: "encryptionkey@0"
I20260929 02:06:29.458063 17377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:29.477064 17377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:29.477967 17377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:29.478281 17377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:29.498629 17377 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:46195
I20260929 02:06:29.498656 17442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:46195 every 8 connection(s)
I20260929 02:06:29.500088 17377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:29.504048 17443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:29.504153 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 17377
I20260929 02:06:29.504421 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:29.505045 11995 external_mini_cluster.cc:1428] Setting key afbe3aab019ae28b01a19ec9c6eadc86
I20260929 02:06:29.519855 17443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01: Bootstrap starting.
I20260929 02:06:29.521967 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:29.507262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.11.182.254:46195)
0929 02:06:29.508454 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:29.508470 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:29.508521 (+ 51us) server_negotiation.cc:407] Connection header received
0929 02:06:29.510994 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:29.511029 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:29.511754 (+ 725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:29.512257 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:29.512861 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:29.514823 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:29.516425 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:29.517603 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:29.517800 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:29.519037 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:29.519049 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:29.519073 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:29.519863 (+ 790us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:29.519982 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:29.520008 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:29.520122 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:29.520295 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 02:06:29.520632 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":354,"threads_started":1}
I20260929 02:06:29.523830 17443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:29.525086 17443 log.cc:824] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:29.526446 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:06:29.528434 17443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01: No bootstrap required, opened a new log
I20260929 02:06:29.536401 17443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1c69c2ae640bd9342fdd944809a01" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46195 } }
I20260929 02:06:29.536842 17443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:29.536949 17443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a1c69c2ae640bd9342fdd944809a01, State: Initialized, Role: FOLLOWER
I20260929 02:06:29.537617 17443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1c69c2ae640bd9342fdd944809a01" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46195 } }
I20260929 02:06:29.537868 17443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:29.538013 17443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:29.538190 17443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:29.540766 17443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1c69c2ae640bd9342fdd944809a01" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46195 } }
I20260929 02:06:29.541271 17443 leader_election.cc:302] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a1c69c2ae640bd9342fdd944809a01; no voters:
I20260929 02:06:29.541865 17443 leader_election.cc:288] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:29.542021 17448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:29.542825 17448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [term 1 LEADER]: Becoming Leader. State: Replica: 64a1c69c2ae640bd9342fdd944809a01, State: Running, Role: LEADER
I20260929 02:06:29.543454 17448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1c69c2ae640bd9342fdd944809a01" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46195 } }
I20260929 02:06:29.544179 17443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:29.546464 17449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a1c69c2ae640bd9342fdd944809a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1c69c2ae640bd9342fdd944809a01" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46195 } } }
I20260929 02:06:29.546445 17450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a1c69c2ae640bd9342fdd944809a01. Latest consensus state: current_term: 1 leader_uuid: "64a1c69c2ae640bd9342fdd944809a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1c69c2ae640bd9342fdd944809a01" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46195 } } }
I20260929 02:06:29.547003 17449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:29.547043 17450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:29.547994 17454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:29.554380 17454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:29.564317 17454 catalog_manager.cc:1347] Generated new cluster ID: f8239d0c83364196bd3efbbf79058204
I20260929 02:06:29.564495 17454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:29.576504 17454 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:29.577913 17454 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:29.589784 17454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a1c69c2ae640bd9342fdd944809a01: Generated new TSK 0
I20260929 02:06:29.591070 17454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:06:29.627856 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:29.615960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.11.182.254:46195)
0929 02:06:29.616128 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:29.616133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:29.619443 (+ 3310us) server_negotiation.cc:407] Connection header received
0929 02:06:29.619708 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:29.619712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:29.619788 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:29.619978 (+ 190us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:06:29.620517 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:29.621332 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:29.622769 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:29.623539 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:29.623764 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:29.624370 (+ 606us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:06:29.627173 (+ 2803us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:06:29.627538 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40006: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 02:06:29.628159 17472 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:29.615842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46195 (local address 127.0.0.1:40006)
0929 02:06:29.619291 (+ 3449us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:29.619329 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:29.619518 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:29.619996 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:29.620001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:29.620013 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:06:29.620355 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:29.620365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:29.621496 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:29.621503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:29.622609 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:29.622621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:29.623842 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:29.623856 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:29.624014 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:29.624053 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:06:29.627425 (+ 3372us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:06:29.627739 (+ 314us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.182.254:46195: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":3326,"thread_start_us":3198,"threads_started":1}
W20260929 02:06:29.628212 17446 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40006: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260929 02:06:29.629820 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 17377
I20260929 02:06:29.631088 17438 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 02:06:29.645347 17473 webserver.cc:533] Webserver: sslize failed:
E20260929 02:06:29.645538 17473 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:06:29.728863 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 17377
2026-09-29T02:06:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (775 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T02:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:29Z Disabled control of system clock
I20260929 02:06:29.813441 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:36989
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:36989
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/testchainca.cert with env {}
W20260929 02:06:30.164052 17478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:30.164513 17478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:30.164651 17478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:30.172562 17478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:30.172689 17478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:30.172729 17478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:30.172787 17478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:30.172832 17478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:06:30.172888 17478 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:06:30.172927 17478 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 02:06:30.183635 17478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:36989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:36989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:30.186316 17478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:30.187816 17478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:30.200587 17484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:30.200577 17486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:30.200577 17483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:30.201536 17478 server_base.cc:1034] running on GCE node
I20260929 02:06:30.202392 17478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:30.204473 17478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:30.205724 17478 hybrid_clock.cc:648] HybridClock initialized: now 1790647590205677 us; error 48 us; skew 500 ppm
I20260929 02:06:30.206127 17478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:06:30.208693 17478 webserver.cc:460] Webserver started at http://127.11.182.254:35137/ using document root <none> and password file <none>
I20260929 02:06:30.209436 17478 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:30.209570 17478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:30.209908 17478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:30.212879 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "22ae460db6cf45d98ef2a52529a3fd1a"
format_stamp: "Formatted at 2026-09-29 02:06:30 on dist-test-slave-q06p"
server_key: "d7a0881ae741ece19fa4de15f4d61cfb"
server_key_iv: "2f272d51a9b232e25dc933e0efa98a48"
server_key_version: "encryptionkey@0"
I20260929 02:06:30.213718 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "22ae460db6cf45d98ef2a52529a3fd1a"
format_stamp: "Formatted at 2026-09-29 02:06:30 on dist-test-slave-q06p"
server_key: "d7a0881ae741ece19fa4de15f4d61cfb"
server_key_iv: "2f272d51a9b232e25dc933e0efa98a48"
server_key_version: "encryptionkey@0"
I20260929 02:06:30.219588 17478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:06:30.223443 17492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:30.225023 17478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 02:06:30.225340 17478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "22ae460db6cf45d98ef2a52529a3fd1a"
format_stamp: "Formatted at 2026-09-29 02:06:30 on dist-test-slave-q06p"
server_key: "d7a0881ae741ece19fa4de15f4d61cfb"
server_key_iv: "2f272d51a9b232e25dc933e0efa98a48"
server_key_version: "encryptionkey@0"
I20260929 02:06:30.225589 17478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:30.242017 17478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:30.242993 17478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:30.243281 17478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:30.263087 17478 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:36989
I20260929 02:06:30.263108 17543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:36989 every 8 connection(s)
I20260929 02:06:30.264521 17478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:30.266412 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 17478
I20260929 02:06:30.266618 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:30.267478 11995 external_mini_cluster.cc:1428] Setting key fd8aa230cd6bc6cbb58ef43fdefc36d1
I20260929 02:06:30.269963 17544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:30.283257 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:30.269610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.11.182.254:36989)
0929 02:06:30.270722 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:30.270734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:30.270781 (+ 47us) server_negotiation.cc:407] Connection header received
0929 02:06:30.272558 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:30.272590 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:30.273170 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:30.273623 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:30.274499 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:30.276183 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:30.278195 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:30.279104 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:30.279268 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:30.280660 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:30.280668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:30.280685 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:06:30.281300 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:30.281386 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:06:30.281412 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:30.281507 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:30.281754 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 02:06:30.282191 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":424,"threads_started":1}
I20260929 02:06:30.283962 17544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a: Bootstrap starting.
I20260929 02:06:30.287657 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:06:30.288053 17544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:30.289155 17544 log.cc:824] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:30.292529 17544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a: No bootstrap required, opened a new log
I20260929 02:06:30.300763 17544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae460db6cf45d98ef2a52529a3fd1a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36989 } }
I20260929 02:06:30.301143 17544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:30.301216 17544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ae460db6cf45d98ef2a52529a3fd1a, State: Initialized, Role: FOLLOWER
I20260929 02:06:30.301942 17544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae460db6cf45d98ef2a52529a3fd1a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36989 } }
I20260929 02:06:30.302201 17549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:30.302471 17549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae460db6cf45d98ef2a52529a3fd1a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36989 } }
I20260929 02:06:30.303141 17549 leader_election.cc:302] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ae460db6cf45d98ef2a52529a3fd1a; no voters:
I20260929 02:06:30.303782 17544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:30.303966 17544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:30.303974 17549 leader_election.cc:288] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260929 02:06:30.304111 17544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:30.306900 17544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae460db6cf45d98ef2a52529a3fd1a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36989 } }
I20260929 02:06:30.307185 17544 leader_election.cc:302] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ae460db6cf45d98ef2a52529a3fd1a; no voters:
I20260929 02:06:30.307507 17544 leader_election.cc:288] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:30.307518 17550 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 02:06:30.308444 17549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:30.308310 17551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae460db6cf45d98ef2a52529a3fd1a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36989 } } }
I20260929 02:06:30.308655 17551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:06:30.308609 17549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [term 1 LEADER]: Becoming Leader. State: Replica: 22ae460db6cf45d98ef2a52529a3fd1a, State: Running, Role: LEADER
I20260929 02:06:30.309255 17549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae460db6cf45d98ef2a52529a3fd1a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36989 } }
I20260929 02:06:30.309536 17544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:30.311246 17550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ae460db6cf45d98ef2a52529a3fd1a. Latest consensus state: current_term: 1 leader_uuid: "22ae460db6cf45d98ef2a52529a3fd1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae460db6cf45d98ef2a52529a3fd1a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36989 } } }
I20260929 02:06:30.311652 17550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:30.312713 17555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:30.321734 17555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:30.331862 17555 catalog_manager.cc:1347] Generated new cluster ID: 1025b60ed2fa4f29af53209b3119c70d
I20260929 02:06:30.332029 17555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:30.347128 17555 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:30.348469 17555 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:30.358284 17555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22ae460db6cf45d98ef2a52529a3fd1a: Generated new TSK 0
I20260929 02:06:30.359254 17555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:06:30.407687 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:30.382646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.11.182.254:36989)
0929 02:06:30.382876 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:30.382883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:30.383192 (+ 309us) server_negotiation.cc:407] Connection header received
0929 02:06:30.383409 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:30.383415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:30.383506 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:30.383688 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:06:30.384172 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:30.385468 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:30.386948 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:30.387713 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:30.387851 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:30.388330 (+ 479us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:06:30.406918 (+ 18588us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:06:30.407364 (+ 446us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39684: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 02:06:30.407835 17547 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39684: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 02:06:30.407850 17573 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:30.382478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36989 (local address 127.0.0.1:39684)
0929 02:06:30.383091 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:30.383109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:30.383238 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:30.383730 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:30.383734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:30.383744 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:06:30.383990 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:30.383998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:30.385618 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:30.385623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:30.386750 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:30.386761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:30.387882 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:30.387887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:30.388011 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:30.388045 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:06:30.407234 (+ 19189us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:06:30.407515 (+ 281us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.182.254:36989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":312,"threads_started":1}
I20260929 02:06:30.409579 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 17478
I20260929 02:06:30.410992 17539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:30.510912 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 17478
2026-09-29T02:06:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (759 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (433 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (452 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:31 dist-test-slave-q06p krb5kdc[17588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:31 dist-test-slave-q06p krb5kdc[17588](info): set up 2 sockets
Sep 29 02:06:31 dist-test-slave-q06p krb5kdc[17588](info): commencing operation
krb5kdc: starting...
W20260929 02:06:32.504140 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:32 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:06:32.715437 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:46053
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:46053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:33.037688 17601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:33.037993 17601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:33.038142 17601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:33.045475 17601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:33.045588 17601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:33.045688 17601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:33.045778 17601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:33.045849 17601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:33.056016 17601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:46053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:46053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:33.057381 17601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:33.058671 17601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:33.070736 17606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:33.070782 17609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:33.071058 17607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:33.071923 17601 server_base.cc:1034] running on GCE node
I20260929 02:06:33.072770 17601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:33.074282 17601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:33.075501 17601 hybrid_clock.cc:648] HybridClock initialized: now 1790647593075442 us; error 53 us; skew 500 ppm
I20260929 02:06:33.075846 17601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:33 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647593, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:33.083284 17601 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:06:33.084857 17601 webserver.cc:460] Webserver started at http://127.11.182.254:35423/ using document root <none> and password file <none>
I20260929 02:06:33.085593 17601 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:33.085722 17601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:33.086041 17601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:33.089026 17601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "c678a4a102e143488033476ec8b6754e"
format_stamp: "Formatted at 2026-09-29 02:06:33 on dist-test-slave-q06p"
server_key: "1ad4e81bde6148714f22b09a1028136b"
server_key_iv: "ae08452497bc77f3fd7e0abc571284fd"
server_key_version: "encryptionkey@0"
I20260929 02:06:33.089779 17601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "c678a4a102e143488033476ec8b6754e"
format_stamp: "Formatted at 2026-09-29 02:06:33 on dist-test-slave-q06p"
server_key: "1ad4e81bde6148714f22b09a1028136b"
server_key_iv: "ae08452497bc77f3fd7e0abc571284fd"
server_key_version: "encryptionkey@0"
I20260929 02:06:33.095193 17601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:06:33.098558 17616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:33.099943 17601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:33.100121 17601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "c678a4a102e143488033476ec8b6754e"
format_stamp: "Formatted at 2026-09-29 02:06:33 on dist-test-slave-q06p"
server_key: "1ad4e81bde6148714f22b09a1028136b"
server_key_iv: "ae08452497bc77f3fd7e0abc571284fd"
server_key_version: "encryptionkey@0"
I20260929 02:06:33.100329 17601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:33.117486 17601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:33.121402 17601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:33.121697 17601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:33.139412 17601 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:46053
I20260929 02:06:33.139436 17667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:46053 every 8 connection(s)
I20260929 02:06:33.140798 17601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:33.144577 17668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:33.145072 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 17601
I20260929 02:06:33.145267 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:33.145893 11995 external_mini_cluster.cc:1428] Setting key 30fec231f44b625b65089ab03a023941
I20260929 02:06:33.158630 17668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e: Bootstrap starting.
Sep 29 02:06:33 dist-test-slave-q06p krb5kdc[17588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:33.161761 17668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:33.163110 17668 log.cc:824] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:33.166323 17668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e: No bootstrap required, opened a new log
I20260929 02:06:33.169821 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:33.147975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.11.182.254:46053)
0929 02:06:33.148757 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:33.148773 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:33.148825 (+ 52us) server_negotiation.cc:407] Connection header received
0929 02:06:33.151066 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:33.151095 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:33.151622 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:33.152086 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:33.153307 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:33.155329 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:33.156530 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:33.157562 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:33.157732 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:33.161540 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:33.161554 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:33.161574 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:33.161626 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:33.165332 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:33.165990 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:33.165998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:33.166006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:33.166203 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:33.166568 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:33.166575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:33.166579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:33.167319 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:33.167603 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:33.167982 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 02:06:33.168433 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":324,"threads_started":1}
I20260929 02:06:33.175414 17668 raft_consensus.cc:348] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c678a4a102e143488033476ec8b6754e" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46053 } }
I20260929 02:06:33.175752 17668 raft_consensus.cc:374] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:33.175823 17668 raft_consensus.cc:729] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c678a4a102e143488033476ec8b6754e, State: Initialized, Role: FOLLOWER
I20260929 02:06:33.176302 17668 consensus_queue.cc:260] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c678a4a102e143488033476ec8b6754e" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46053 } }
I20260929 02:06:33.176587 17668 raft_consensus.cc:388] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:33.176756 17668 raft_consensus.cc:482] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:33.176936 17668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:33.179114 17668 raft_consensus.cc:504] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c678a4a102e143488033476ec8b6754e" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46053 } }
I20260929 02:06:33.179603 17668 leader_election.cc:302] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c678a4a102e143488033476ec8b6754e; no voters:
I20260929 02:06:33.180189 17668 leader_election.cc:288] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:33.180366 17673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:33.181516 17673 raft_consensus.cc:686] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [term 1 LEADER]: Becoming Leader. State: Replica: c678a4a102e143488033476ec8b6754e, State: Running, Role: LEADER
I20260929 02:06:33.182041 17673 consensus_queue.cc:237] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c678a4a102e143488033476ec8b6754e" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46053 } }
I20260929 02:06:33.182298 17668 sys_catalog.cc:564] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:33.185258 17674 sys_catalog.cc:455] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c678a4a102e143488033476ec8b6754e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c678a4a102e143488033476ec8b6754e" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46053 } } }
I20260929 02:06:33.185700 17674 sys_catalog.cc:458] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:33.185201 17675 sys_catalog.cc:455] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c678a4a102e143488033476ec8b6754e. Latest consensus state: current_term: 1 leader_uuid: "c678a4a102e143488033476ec8b6754e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c678a4a102e143488033476ec8b6754e" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46053 } } }
I20260929 02:06:33.186172 17675 sys_catalog.cc:458] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:33.189527 17682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:33.195138 17682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:33.203285 17682 catalog_manager.cc:1347] Generated new cluster ID: adc34ac308634643a1335629ecfd71de
I20260929 02:06:33.203573 17682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:33.243306 17682 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:33.244712 17682 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:33.272887 17682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c678a4a102e143488033476ec8b6754e: Generated new TSK 0
I20260929 02:06:33.274031 17682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:06:33.332304 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46053
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:33.663205 17694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:33.663486 17694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:33.663640 17694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:33.671334 17694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:33.671450 17694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:33.671623 17694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:33.681393 17694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:33.682787 17694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:33.684193 17694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:33.697860 17699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:33.697899 17700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:33.697922 17702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:33.700317 17694 server_base.cc:1034] running on GCE node
I20260929 02:06:33.700918 17694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:33.701885 17694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:33.703076 17694 hybrid_clock.cc:648] HybridClock initialized: now 1790647593703029 us; error 35 us; skew 500 ppm
I20260929 02:06:33.703449 17694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:33 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647593, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:33.706588 17694 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:06:33.708104 17694 webserver.cc:460] Webserver started at http://127.11.182.193:40673/ using document root <none> and password file <none>
I20260929 02:06:33.708778 17694 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:33.708875 17694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:33.709123 17694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:33.712076 17694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "90b645c88d6e49d2b62129f88e325e98"
format_stamp: "Formatted at 2026-09-29 02:06:33 on dist-test-slave-q06p"
server_key: "acdf594df287f8f3a9c9788ee48a3f8a"
server_key_iv: "943d2a689eb902f8a5a4c28962e088e2"
server_key_version: "encryptionkey@0"
I20260929 02:06:33.712810 17694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "90b645c88d6e49d2b62129f88e325e98"
format_stamp: "Formatted at 2026-09-29 02:06:33 on dist-test-slave-q06p"
server_key: "acdf594df287f8f3a9c9788ee48a3f8a"
server_key_iv: "943d2a689eb902f8a5a4c28962e088e2"
server_key_version: "encryptionkey@0"
I20260929 02:06:33.717830 17694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 02:06:33.721212 17709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:33.722488 17694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:33.722649 17694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "90b645c88d6e49d2b62129f88e325e98"
format_stamp: "Formatted at 2026-09-29 02:06:33 on dist-test-slave-q06p"
server_key: "acdf594df287f8f3a9c9788ee48a3f8a"
server_key_iv: "943d2a689eb902f8a5a4c28962e088e2"
server_key_version: "encryptionkey@0"
I20260929 02:06:33.722847 17694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:33.738334 17694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:33.742650 17694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:33.743018 17694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:33.743965 17694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:33.745244 17694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:33.745340 17694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:33.745477 17694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:33.745546 17694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:33.777878 17694 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:32913
I20260929 02:06:33.777935 17821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:32913 every 8 connection(s)
I20260929 02:06:33.779219 17694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:33.781977 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 17694
I20260929 02:06:33.782194 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:33.782936 11995 external_mini_cluster.cc:1428] Setting key 86f57367d8add2d983e352a4cea015a0
Sep 29 02:06:33 dist-test-slave-q06p krb5kdc[17588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647593, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:33.800166 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:33.781630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:57381 (local address 127.11.182.254:46053)
0929 02:06:33.782124 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:33.782131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:33.783352 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 02:06:33.785135 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:33.785141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:33.785226 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:33.785338 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:33.787080 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:33.788225 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:33.789750 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:33.790466 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:33.790611 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:33.794636 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:33.794641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:33.794644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:33.794677 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:33.797126 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:33.797852 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:33.797861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:33.797865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:33.797974 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:33.798388 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:33.798393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:33.798396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:33.798722 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:33.798902 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:33.799587 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 02:06:33.799804 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":229,"threads_started":1}
I20260929 02:06:33.801136 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:33.782256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46053 (local address 127.11.182.193:57381)
0929 02:06:33.783174 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:33.783236 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:33.784888 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:33.785605 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:33.785622 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:33.786167 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:33.786900 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:33.786921 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:33.788386 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:33.788390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:33.789613 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:33.789624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:33.790674 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:33.790678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:33.791055 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:33.791771 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:33.791827 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:33.794408 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:33.797277 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:33.797286 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:33.797302 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:33.797726 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:33.798079 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:33.798085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:33.798089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:33.798272 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:33.798873 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:33.798881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:33.799216 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 02:06:33.799484 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 02:06:33.800017 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":367,"threads_started":1}
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:06:33.802884 17822 heartbeater.cc:347] Connected to a master server at 127.11.182.254:46053
I20260929 02:06:33.803180 17822 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:33.803933 17822 heartbeater.cc:511] Master 127.11.182.254:46053 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
I20260929 02:06:33.806142 17633 ts_manager.cc:194] Registered new tserver with Master: 90b645c88d6e49d2b62129f88e325e98 (127.11.182.193:32913)
I20260929 02:06:33.807605 17633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:57381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:06:33.829455 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46053
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:34.154692 17828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:34.154970 17828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:34.155149 17828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:34.162290 17828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:34.162411 17828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:34.162590 17828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:34.172670 17828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:34.173911 17828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:34.175277 17828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:34.189080 17834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:34.189020 17836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:34.189178 17833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:34.189934 17828 server_base.cc:1034] running on GCE node
I20260929 02:06:34.190532 17828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:34.191532 17828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:34.192739 17828 hybrid_clock.cc:648] HybridClock initialized: now 1790647594192674 us; error 55 us; skew 500 ppm
I20260929 02:06:34.193073 17828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:34.196228 17828 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:06:34.197590 17828 webserver.cc:460] Webserver started at http://127.11.182.194:34013/ using document root <none> and password file <none>
I20260929 02:06:34.198311 17828 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:34.198463 17828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:34.198769 17828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:34.201680 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "e8c32a21420f446a96f09f8bc0563f73"
format_stamp: "Formatted at 2026-09-29 02:06:34 on dist-test-slave-q06p"
server_key: "9633257f47ac16948e19f15d970dba8f"
server_key_iv: "88d09f75116ec35da017f7e57d2e6203"
server_key_version: "encryptionkey@0"
I20260929 02:06:34.202464 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "e8c32a21420f446a96f09f8bc0563f73"
format_stamp: "Formatted at 2026-09-29 02:06:34 on dist-test-slave-q06p"
server_key: "9633257f47ac16948e19f15d970dba8f"
server_key_iv: "88d09f75116ec35da017f7e57d2e6203"
server_key_version: "encryptionkey@0"
I20260929 02:06:34.207569 17828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:06:34.210898 17843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:34.212168 17828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:06:34.212342 17828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "e8c32a21420f446a96f09f8bc0563f73"
format_stamp: "Formatted at 2026-09-29 02:06:34 on dist-test-slave-q06p"
server_key: "9633257f47ac16948e19f15d970dba8f"
server_key_iv: "88d09f75116ec35da017f7e57d2e6203"
server_key_version: "encryptionkey@0"
I20260929 02:06:34.212535 17828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:34.227435 17828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:34.231441 17828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:34.231706 17828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:34.232590 17828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:34.233821 17828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:34.233889 17828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:34.234041 17828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:34.234089 17828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:34.265278 17828 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:35873
I20260929 02:06:34.265349 17955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:35873 every 8 connection(s)
I20260929 02:06:34.266517 17828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:34.269892 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 17828
I20260929 02:06:34.270094 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:34.270725 11995 external_mini_cluster.cc:1428] Setting key bc190f556d863cbea433db77bd2790a5
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:34.288796 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:34.269051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:41847 (local address 127.11.182.254:46053)
0929 02:06:34.269212 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.269216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.270459 (+ 1243us) server_negotiation.cc:407] Connection header received
0929 02:06:34.272605 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.272612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.272720 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.272833 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.274948 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.275620 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.278135 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.278526 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.278614 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:34.282890 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:34.282903 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:34.282906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:34.282941 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:34.285618 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.286408 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.286418 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.286421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.286539 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.287055 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.287060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.287063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.287406 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:34.287565 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:34.288235 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 02:06:34.288464 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:06:34.289767 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:34.269378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46053 (local address 127.11.182.194:41847)
0929 02:06:34.270258 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:34.270322 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:34.272293 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:34.273100 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:34.273112 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:34.273848 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:34.274706 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:34.274735 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.275782 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.275791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:34.277989 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:34.278008 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.278751 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.278761 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:34.279149 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:34.279903 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:34.279938 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:34.282613 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:34.285809 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:34.285818 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:34.285835 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:34.286290 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:34.286653 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:34.286659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:34.286663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:34.286945 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:34.287529 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:34.287537 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:34.287882 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 02:06:34.288136 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 02:06:34.288654 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":335,"threads_started":1}
I20260929 02:06:34.291325 17956 heartbeater.cc:347] Connected to a master server at 127.11.182.254:46053
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:06:34.291623 17956 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:34.292325 17956 heartbeater.cc:511] Master 127.11.182.254:46053 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
I20260929 02:06:34.293853 17633 ts_manager.cc:194] Registered new tserver with Master: e8c32a21420f446a96f09f8bc0563f73 (127.11.182.194:35873)
I20260929 02:06:34.294631 17633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:41847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:06:34.317837 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46053
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:34.634895 17961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:34.635208 17961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:34.635370 17961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:34.642375 17961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:34.642516 17961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:34.642699 17961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:34.652022 17961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:34.653276 17961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:34.654553 17961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:34.667165 17969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:34.667143 17966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:34.667110 17967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:34.667869 17961 server_base.cc:1034] running on GCE node
I20260929 02:06:34.668460 17961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:34.669384 17961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:34.670606 17961 hybrid_clock.cc:648] HybridClock initialized: now 1790647594670554 us; error 44 us; skew 500 ppm
I20260929 02:06:34.670969 17961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:34.673975 17961 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:06:34.675367 17961 webserver.cc:460] Webserver started at http://127.11.182.195:34249/ using document root <none> and password file <none>
I20260929 02:06:34.676026 17961 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:34.676169 17961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:34.676468 17961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:34.679322 17961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "de80d9165a06478e8ae5a9bb082f9188"
format_stamp: "Formatted at 2026-09-29 02:06:34 on dist-test-slave-q06p"
server_key: "99ade2e6507db1fc2dc92e2e14b86bf8"
server_key_iv: "83a879bbcf2fe88a744b8078ed024538"
server_key_version: "encryptionkey@0"
I20260929 02:06:34.680060 17961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "de80d9165a06478e8ae5a9bb082f9188"
format_stamp: "Formatted at 2026-09-29 02:06:34 on dist-test-slave-q06p"
server_key: "99ade2e6507db1fc2dc92e2e14b86bf8"
server_key_iv: "83a879bbcf2fe88a744b8078ed024538"
server_key_version: "encryptionkey@0"
I20260929 02:06:34.685160 17961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:06:34.688661 17976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:34.689893 17961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:06:34.690056 17961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "de80d9165a06478e8ae5a9bb082f9188"
format_stamp: "Formatted at 2026-09-29 02:06:34 on dist-test-slave-q06p"
server_key: "99ade2e6507db1fc2dc92e2e14b86bf8"
server_key_iv: "83a879bbcf2fe88a744b8078ed024538"
server_key_version: "encryptionkey@0"
I20260929 02:06:34.690263 17961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:34.712990 17961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:34.717463 17961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:34.717752 17961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:34.718685 17961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:34.719996 17961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:34.720108 17961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:34.720248 17961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:34.720319 17961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:34.752717 17961 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:39701
I20260929 02:06:34.752799 18088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:39701 every 8 connection(s)
I20260929 02:06:34.753960 17961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:34.755192 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 17961
I20260929 02:06:34.755378 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:34.755931 11995 external_mini_cluster.cc:1428] Setting key b387c8cc7a579bd607e304043e9241d2
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:34.773345 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:34.756701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:41677 (local address 127.11.182.254:46053)
0929 02:06:34.756911 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.756916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.758190 (+ 1274us) server_negotiation.cc:407] Connection header received
0929 02:06:34.760177 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.760182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.760281 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.760387 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.761811 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.762547 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.764186 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.764599 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.764726 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:34.768073 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:34.768078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:34.768081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:34.768110 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:34.770691 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.771462 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.771466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.771468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.771587 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.771952 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.771955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.771957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.772205 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:34.772327 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:34.772946 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 02:06:34.773094 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:06:34.774327 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:34.757207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46053 (local address 127.11.182.195:41677)
0929 02:06:34.758037 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:34.758079 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:34.759928 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:34.760616 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:34.760624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:34.761108 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:34.761658 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:34.761674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.762680 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.762685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:34.764044 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:34.764056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.764826 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.764833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:34.765072 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:34.765753 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:34.765774 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:34.767866 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:34.770815 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:34.770822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:34.770835 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:34.771353 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:34.771673 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:34.771677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:34.771679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:34.771860 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:34.772309 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:34.772314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:34.772612 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 02:06:34.772853 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 02:06:34.773227 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":310,"threads_started":1}
I20260929 02:06:34.775658 18089 heartbeater.cc:347] Connected to a master server at 127.11.182.254:46053
I20260929 02:06:34.775925 18089 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:34.776563 18089 heartbeater.cc:511] Master 127.11.182.254:46053 requested a full tablet report, sending...
I20260929 02:06:34.777882 17633 ts_manager.cc:194] Registered new tserver with Master: de80d9165a06478e8ae5a9bb082f9188 (127.11.182.195:39701)
I20260929 02:06:34.778697 17633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:41677
I20260929 02:06:34.783387 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/test-admin.keytab.
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:34.809172 11995 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260929 02:06:34.810387 17822 heartbeater.cc:503] Master 127.11.182.254:46053 was elected leader, sending a full tablet report...
W20260929 02:06:34.814193 18096 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:34.811807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.11.182.193:32913)
0929 02:06:34.812386 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.812393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.812441 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:06:34.812544 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.812551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.812774 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.812898 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.813752 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43594: BlockingRecv error: recv got EOF from 127.0.0.1:43594 (error 108)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":268,"threads_started":1}
W20260929 02:06:34.814177 11995 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:34.815840 11995 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:34.817365 11995 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:06:34.831777 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:34.818977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.11.182.193:32913)
0929 02:06:34.819154 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.819160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.819355 (+ 195us) server_negotiation.cc:407] Connection header received
0929 02:06:34.819486 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.819489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.819544 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.819622 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.820546 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.821495 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.822662 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.823259 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.823362 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:34.826837 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:34.826879 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:34.826898 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:34.826933 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:34.829506 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.830105 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.830111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.830113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.830215 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.830512 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.830517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.830520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.830895 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:34.831117 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:34.831337 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 02:06:34.831516 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 02:06:34.837150 18096 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:34.835624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.11.182.193:32913)
0929 02:06:34.835843 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.835848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.836036 (+ 188us) server_negotiation.cc:407] Connection header received
0929 02:06:34.836173 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.836176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.836240 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.836312 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.836930 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43606: BlockingRecv error: recv got EOF from 127.0.0.1:43606 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 02:06:34.837261 11995 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:34.838675 11995 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:34.839943 11995 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:06:34.852659 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:34.841187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.11.182.193:32913)
0929 02:06:34.841381 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.841388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.841570 (+ 182us) server_negotiation.cc:407] Connection header received
0929 02:06:34.841748 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.841752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.841803 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.841894 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.842765 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.843510 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.844687 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.845102 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.845230 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:34.848554 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:34.848558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:34.848561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:34.848599 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:34.850758 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.851333 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.851337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.851339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.851444 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.851730 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.851733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.851735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.851957 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:34.852134 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:34.852320 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 02:06:34.852454 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 02:06:34.856379 18096 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:34.855065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.11.182.193:32913)
0929 02:06:34.855234 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.855246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.855343 (+ 97us) server_negotiation.cc:407] Connection header received
0929 02:06:34.855452 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.855455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.855514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.855585 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.856180 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43622: BlockingRecv error: recv got EOF from 127.0.0.1:43622 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 02:06:34.856524 11995 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647540175441-11995-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:34.857862 11995 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:34.859154 11995 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:06:34.872032 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:34.860287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.11.182.193:32913)
0929 02:06:34.860480 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.860486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.860694 (+ 208us) server_negotiation.cc:407] Connection header received
0929 02:06:34.860801 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.860804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.860852 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.860925 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.861968 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.862640 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.863794 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.864175 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.864264 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:34.867563 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:34.867567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:34.867570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:34.867596 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:34.869907 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.870447 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.870450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.870452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.870584 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.870905 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.870909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.870911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.871132 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:34.871353 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:34.871542 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 02:06:34.871714 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:06:34.873507 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 17694
I20260929 02:06:34.884738 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:34.874143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.11.182.193:32913)
0929 02:06:34.874288 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.874291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.874647 (+ 356us) server_negotiation.cc:407] Connection header received
0929 02:06:34.874815 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.874818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.874935 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.875006 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.876004 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.876632 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.877907 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.878311 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.878399 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:34.880345 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:34.880349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:34.880352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:34.880377 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:34.882614 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.883224 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.883228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.883230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.883397 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.883720 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.883723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.883725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.883942 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:34.884123 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:34.884357 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 02:06:34.884499 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:06:34.885263 17817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:34.967478 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 17694
I20260929 02:06:34.983222 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 17828
Sep 29 02:06:34 dist-test-slave-q06p krb5kdc[17588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:06:34.997653 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:34.984119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.11.182.194:35873)
0929 02:06:34.984651 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:34.984658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:34.984681 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:06:34.984738 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:34.984744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:34.984951 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:34.985081 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:34.986026 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.987135 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.988266 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:34.988863 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:34.988952 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:34.992503 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:34.992513 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:34.992526 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:34.992558 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:34.995376 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.995966 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.995973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.995975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.996085 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:34.996398 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:34.996401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:34.996403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:34.996714 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:34.996932 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:34.997179 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 02:06:34.997371 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":295,"threads_started":1}
I20260929 02:06:34.999066 17951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:35.079840 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 17828
I20260929 02:06:35.096002 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 17961
Sep 29 02:06:35 dist-test-slave-q06p krb5kdc[17588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:06:35.111054 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:35.096891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.11.182.195:39701)
0929 02:06:35.097525 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:35.097532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:35.097558 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:06:35.097615 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:35.097621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:35.097824 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:35.097983 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:35.099104 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:35.100093 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:35.101293 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:35.101819 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:35.101951 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:35.105750 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:35.105760 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:35.105775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:35.105810 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:35.108379 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:35.109059 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:35.109066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:35.109068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:35.109183 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:35.109540 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:35.109545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:35.109549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:35.109882 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:35.110201 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:35.110446 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 02:06:35.110695 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":338,"threads_started":1}
I20260929 02:06:35.112314 18084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:35.200484 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 17961
I20260929 02:06:35.216030 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 17601
I20260929 02:06:35.217166 17663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:35.322090 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 17601
2026-09-29T02:06:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3938 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:35 dist-test-slave-q06p krb5kdc[18116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:35 dist-test-slave-q06p krb5kdc[18116](info): set up 2 sockets
Sep 29 02:06:35 dist-test-slave-q06p krb5kdc[18116](info): commencing operation
krb5kdc: starting...
W20260929 02:06:36.420043 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:36 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.254.keytab.
Entry for principal oryx/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.254.keytab.
I20260929 02:06:36.631904 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:44915
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.254.keytab
--principal=oryx/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:44915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:36.946998 18129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:36.947269 18129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:36.947357 18129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:36.954464 18129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:36.954567 18129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:36.954613 18129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:36.954660 18129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:36.954710 18129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:36.964318 18129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:44915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.254.keytab
--principal=oryx/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:44915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:36.965529 18129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:36.966743 18129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:36.979306 18137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:36.979357 18134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:36.979254 18135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:36.984495 18129 server_base.cc:1034] running on GCE node
I20260929 02:06:36.985291 18129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:36.986730 18129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:36.987972 18129 hybrid_clock.cc:648] HybridClock initialized: now 1790647596987928 us; error 43 us; skew 500 ppm
I20260929 02:06:36.988327 18129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:36 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:36.991531 18129 init.cc:377] Logged in from keytab as oryx/127.11.182.254@KRBTEST.COM (short username oryx)
I20260929 02:06:36.993059 18129 webserver.cc:460] Webserver started at http://127.11.182.254:38501/ using document root <none> and password file <none>
I20260929 02:06:36.993834 18129 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:36.993971 18129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:36.994272 18129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:36.997272 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "6c40684958ea42ad8974f33559cbcfe8"
format_stamp: "Formatted at 2026-09-29 02:06:36 on dist-test-slave-q06p"
server_key: "daaaba3ab784d448f4ae65a6111ac6f2"
server_key_iv: "0a5454b2c9580f85325fe484e84886ba"
server_key_version: "encryptionkey@0"
I20260929 02:06:36.998013 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "6c40684958ea42ad8974f33559cbcfe8"
format_stamp: "Formatted at 2026-09-29 02:06:36 on dist-test-slave-q06p"
server_key: "daaaba3ab784d448f4ae65a6111ac6f2"
server_key_iv: "0a5454b2c9580f85325fe484e84886ba"
server_key_version: "encryptionkey@0"
I20260929 02:06:37.003470 18129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:06:37.006835 18144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:37.008203 18129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:06:37.008378 18129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "6c40684958ea42ad8974f33559cbcfe8"
format_stamp: "Formatted at 2026-09-29 02:06:36 on dist-test-slave-q06p"
server_key: "daaaba3ab784d448f4ae65a6111ac6f2"
server_key_iv: "0a5454b2c9580f85325fe484e84886ba"
server_key_version: "encryptionkey@0"
I20260929 02:06:37.008567 18129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:37.026703 18129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:37.030661 18129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:37.031005 18129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:37.049492 18129 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:44915
I20260929 02:06:37.049574 18195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:44915 every 8 connection(s)
I20260929 02:06:37.050918 18129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:37.052367 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 18129
I20260929 02:06:37.052565 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:37.053399 11995 external_mini_cluster.cc:1428] Setting key f08090109daefe62de844f8c3b30ecd8
I20260929 02:06:37.055648 18196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:37.066222 18196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: Bootstrap starting.
Sep 29 02:06:37 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:37.069766 18196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:37.070869 18196 log.cc:824] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:37.074283 18196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: No bootstrap required, opened a new log
I20260929 02:06:37.075273 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:37.055501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.11.182.254:44915)
0929 02:06:37.056237 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:37.056248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:37.056297 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:06:37.057880 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:37.057900 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:37.058289 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:37.058668 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:37.060094 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:37.061377 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:37.062550 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:37.063276 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:37.063503 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:37.067676 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:37.067683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:37.067696 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:37.067733 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:37.070669 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:37.071522 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:37.071535 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:37.071545 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:37.071754 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:37.072300 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:37.072308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:37.072311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:37.073106 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:37.073396 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:37.073799 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 02:06:37.074174 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":322,"threads_started":1}
I20260929 02:06:37.082379 18196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } }
I20260929 02:06:37.082733 18196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:37.082836 18196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c40684958ea42ad8974f33559cbcfe8, State: Initialized, Role: FOLLOWER
I20260929 02:06:37.083444 18196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } }
I20260929 02:06:37.083673 18196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:37.083804 18196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:37.083966 18196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:37.086066 18196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } }
I20260929 02:06:37.086534 18196 leader_election.cc:302] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c40684958ea42ad8974f33559cbcfe8; no voters:
I20260929 02:06:37.087209 18196 leader_election.cc:288] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:37.087321 18201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:37.088508 18201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c40684958ea42ad8974f33559cbcfe8, State: Running, Role: LEADER
I20260929 02:06:37.088984 18201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } }
I20260929 02:06:37.089298 18196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:37.091820 18203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c40684958ea42ad8974f33559cbcfe8. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } }
I20260929 02:06:37.092182 18203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:37.091915 18202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } }
I20260929 02:06:37.092314 18202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:37.093132 18209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:37.099915 18209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:37.108570 18209 catalog_manager.cc:1347] Generated new cluster ID: eea07e915f3a4c728db042c3d44580c3
I20260929 02:06:37.108690 18209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:37.120818 18209 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:37.122686 18209 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:37.136065 18209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: Generated new TSK 0
I20260929 02:06:37.136962 18209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab.
Entry for principal oryx/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab.
I20260929 02:06:37.196238 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:44915
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab
--principal=oryx/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:37.518724 18222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:37.519023 18222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:37.519223 18222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:37.526654 18222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:37.526772 18222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:37.526978 18222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:37.536809 18222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab
--principal=oryx/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:44915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:37.538132 18222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:37.539533 18222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:37.552366 18228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:37.552386 18227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:37.553038 18230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:37.553076 18222 server_base.cc:1034] running on GCE node
I20260929 02:06:37.553721 18222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:37.554627 18222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:37.555848 18222 hybrid_clock.cc:648] HybridClock initialized: now 1790647597555783 us; error 87 us; skew 500 ppm
I20260929 02:06:37.556154 18222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:37 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647597, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:37.559276 18222 init.cc:377] Logged in from keytab as oryx/127.11.182.193@KRBTEST.COM (short username oryx)
I20260929 02:06:37.560645 18222 webserver.cc:460] Webserver started at http://127.11.182.193:36249/ using document root <none> and password file <none>
I20260929 02:06:37.561291 18222 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:37.561390 18222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:37.561626 18222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:37.564492 18222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "27c9eede04a541b1b1f2dd6b37a882a1"
format_stamp: "Formatted at 2026-09-29 02:06:37 on dist-test-slave-q06p"
server_key: "5432f2d6ab1a6eb39ace7a1f1f6f7f76"
server_key_iv: "e65a66c6536e719123898e4db136dc08"
server_key_version: "encryptionkey@0"
I20260929 02:06:37.565213 18222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "27c9eede04a541b1b1f2dd6b37a882a1"
format_stamp: "Formatted at 2026-09-29 02:06:37 on dist-test-slave-q06p"
server_key: "5432f2d6ab1a6eb39ace7a1f1f6f7f76"
server_key_iv: "e65a66c6536e719123898e4db136dc08"
server_key_version: "encryptionkey@0"
I20260929 02:06:37.570073 18222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 02:06:37.573444 18237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:37.574819 18222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:37.575049 18222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "27c9eede04a541b1b1f2dd6b37a882a1"
format_stamp: "Formatted at 2026-09-29 02:06:37 on dist-test-slave-q06p"
server_key: "5432f2d6ab1a6eb39ace7a1f1f6f7f76"
server_key_iv: "e65a66c6536e719123898e4db136dc08"
server_key_version: "encryptionkey@0"
I20260929 02:06:37.575237 18222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:37.605132 18222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:37.609980 18222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:37.610227 18222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:37.611224 18222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:37.612542 18222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:37.612620 18222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:37.612708 18222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:37.612751 18222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:37.644850 18222 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:35255
I20260929 02:06:37.644886 18349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:35255 every 8 connection(s)
I20260929 02:06:37.646071 18222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:37.647728 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 18222
I20260929 02:06:37.647913 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:37.648736 11995 external_mini_cluster.cc:1428] Setting key 7e18d8fc81304499b0e450353545555c
Sep 29 02:06:37 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647597, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:37.666396 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:37.648587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:57631 (local address 127.11.182.254:44915)
0929 02:06:37.649014 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:37.649018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:37.650243 (+ 1225us) server_negotiation.cc:407] Connection header received
0929 02:06:37.652225 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:37.652231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:37.652315 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:37.652422 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:37.653846 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:37.654535 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:37.656518 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:37.656902 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:37.657004 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:37.660954 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:37.660959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:37.660964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:37.660995 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:37.663548 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:37.664243 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:37.664247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:37.664249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:37.664366 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:37.664775 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:37.664780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:37.664783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:37.665025 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:37.665169 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:37.665806 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 02:06:37.666025 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":227,"threads_started":1}
I20260929 02:06:37.667389 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:37.649273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.193:57631)
0929 02:06:37.650087 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:37.650136 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:37.651965 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:37.652640 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:37.652648 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:37.653140 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:37.653684 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:37.653700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:37.654677 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:37.654688 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:37.656358 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:37.656384 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:37.657110 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:37.657120 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:37.657441 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:37.658124 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:37.658156 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:37.660745 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:37.663681 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:37.663699 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:37.663716 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:37.664108 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:37.664470 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:37.664475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:37.664479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:37.664655 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:37.665146 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:37.665153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:37.665461 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 02:06:37.665713 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 02:06:37.666221 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":309,"threads_started":1}
WARNING: no policy specified for oryx/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:06:37.668948 18350 heartbeater.cc:347] Connected to a master server at 127.11.182.254:44915
I20260929 02:06:37.669288 18350 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:37.669996 18350 heartbeater.cc:511] Master 127.11.182.254:44915 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.194@KRBTEST.COM" created.
I20260929 02:06:37.672058 18161 ts_manager.cc:194] Registered new tserver with Master: 27c9eede04a541b1b1f2dd6b37a882a1 (127.11.182.193:35255)
I20260929 02:06:37.673450 18161 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:57631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab.
Entry for principal oryx/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab.
I20260929 02:06:37.693996 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:44915
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab
--principal=oryx/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:38.013974 18356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:38.014250 18356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:38.014353 18356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:38.021512 18356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:38.021622 18356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:38.021797 18356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:38.031565 18356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab
--principal=oryx/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:44915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:38.032882 18356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:38.034201 18356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:38.047262 18361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:38.047334 18362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:38.047359 18364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:38.048166 18356 server_base.cc:1034] running on GCE node
I20260929 02:06:38.048848 18356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:38.049855 18356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:38.051066 18356 hybrid_clock.cc:648] HybridClock initialized: now 1790647598051007 us; error 57 us; skew 500 ppm
I20260929 02:06:38.051378 18356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:38 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647598, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:38.054553 18356 init.cc:377] Logged in from keytab as oryx/127.11.182.194@KRBTEST.COM (short username oryx)
I20260929 02:06:38.056082 18356 webserver.cc:460] Webserver started at http://127.11.182.194:42141/ using document root <none> and password file <none>
I20260929 02:06:38.056790 18356 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:38.056928 18356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:38.057224 18356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:38.060144 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "2ed372e6334f4cf58de43c740cbddf2f"
format_stamp: "Formatted at 2026-09-29 02:06:38 on dist-test-slave-q06p"
server_key: "6184875d3e6fb50af853826ac5a4ca05"
server_key_iv: "5121848cf8a84943a7e8792f74bae157"
server_key_version: "encryptionkey@0"
I20260929 02:06:38.060917 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "2ed372e6334f4cf58de43c740cbddf2f"
format_stamp: "Formatted at 2026-09-29 02:06:38 on dist-test-slave-q06p"
server_key: "6184875d3e6fb50af853826ac5a4ca05"
server_key_iv: "5121848cf8a84943a7e8792f74bae157"
server_key_version: "encryptionkey@0"
I20260929 02:06:38.065820 18356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:06:38.069376 18371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:38.070693 18356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:06:38.070933 18356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "2ed372e6334f4cf58de43c740cbddf2f"
format_stamp: "Formatted at 2026-09-29 02:06:38 on dist-test-slave-q06p"
server_key: "6184875d3e6fb50af853826ac5a4ca05"
server_key_iv: "5121848cf8a84943a7e8792f74bae157"
server_key_version: "encryptionkey@0"
I20260929 02:06:38.071133 18356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:38.090436 18356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:38.094625 18356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:38.094985 18356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:38.095999 18356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:38.097337 18356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:38.097440 18356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:38.097581 18356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:38.097652 18356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:38.130934 18356 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:34729
I20260929 02:06:38.130939 18483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:34729 every 8 connection(s)
I20260929 02:06:38.132225 18356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:38.133723 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 18356
I20260929 02:06:38.133898 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:38.134416 11995 external_mini_cluster.cc:1428] Setting key 4baead7714459f20d279a840ef8ee02f
Sep 29 02:06:38 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647598, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:06:38.153142 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:38.134905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:44575 (local address 127.11.182.254:44915)
0929 02:06:38.135112 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:38.135119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:38.136548 (+ 1429us) server_negotiation.cc:407] Connection header received
0929 02:06:38.138414 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:38.138420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:38.138507 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:38.138608 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:38.140136 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:38.140822 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:38.142336 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:38.142743 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:38.142885 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:38.146627 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:38.146633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:38.146637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:38.146669 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:38.149866 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:38.150632 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:38.150643 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:38.150646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:38.150775 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:38.151305 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:38.151310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:38.151314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:38.151600 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:38.151762 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:38.152489 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 02:06:38.152744 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:06:38.154258 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:38.135365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.194:44575)
0929 02:06:38.136235 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:38.136452 (+ 217us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:38.138135 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:38.138899 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:38.138910 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:38.139415 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:38.139972 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:38.139989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:38.140971 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:38.140977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:38.142189 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:38.142201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:38.142978 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:38.142984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:38.143222 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:38.144141 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:38.144167 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:38.146374 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:38.150018 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:38.150029 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:38.150055 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:38.150497 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:38.150945 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:38.150952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:38.150956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:38.151174 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:38.151734 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:38.151742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:38.152110 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 02:06:38.152393 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 02:06:38.152932 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.195@KRBTEST.COM" created.
I20260929 02:06:38.155829 18484 heartbeater.cc:347] Connected to a master server at 127.11.182.254:44915
I20260929 02:06:38.156136 18484 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:38.156854 18484 heartbeater.cc:511] Master 127.11.182.254:44915 requested a full tablet report, sending...
I20260929 02:06:38.158321 18161 ts_manager.cc:194] Registered new tserver with Master: 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729)
I20260929 02:06:38.159170 18161 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:44575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab.
Entry for principal oryx/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab.
I20260929 02:06:38.179628 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:44915
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab
--principal=oryx/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:38.511098 18489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:38.511399 18489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:38.511615 18489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:38.521469 18489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:38.521660 18489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:38.521864 18489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:38.532596 18489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab
--principal=oryx/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:44915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:38.533957 18489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:38.535308 18489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:38.548309 18494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:38.548349 18495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:38.548558 18497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:38.549361 18489 server_base.cc:1034] running on GCE node
I20260929 02:06:38.549966 18489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:38.550980 18489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:38.552202 18489 hybrid_clock.cc:648] HybridClock initialized: now 1790647598552152 us; error 48 us; skew 500 ppm
I20260929 02:06:38.552520 18489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:38 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647598, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:38.555979 18489 init.cc:377] Logged in from keytab as oryx/127.11.182.195@KRBTEST.COM (short username oryx)
I20260929 02:06:38.557588 18489 webserver.cc:460] Webserver started at http://127.11.182.195:38581/ using document root <none> and password file <none>
I20260929 02:06:38.558388 18489 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:38.558542 18489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:38.558924 18489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:38.561915 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "b81b0df2be8b4d5cb7b042ee400726d2"
format_stamp: "Formatted at 2026-09-29 02:06:38 on dist-test-slave-q06p"
server_key: "9aa6cd5a8de1c97bfc8ee0f2fa54444c"
server_key_iv: "1fb8eea92fe780082c47df323996a46d"
server_key_version: "encryptionkey@0"
I20260929 02:06:38.562732 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "b81b0df2be8b4d5cb7b042ee400726d2"
format_stamp: "Formatted at 2026-09-29 02:06:38 on dist-test-slave-q06p"
server_key: "9aa6cd5a8de1c97bfc8ee0f2fa54444c"
server_key_iv: "1fb8eea92fe780082c47df323996a46d"
server_key_version: "encryptionkey@0"
I20260929 02:06:38.568477 18489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:06:38.572193 18504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:38.573814 18489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:06:38.574048 18489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "b81b0df2be8b4d5cb7b042ee400726d2"
format_stamp: "Formatted at 2026-09-29 02:06:38 on dist-test-slave-q06p"
server_key: "9aa6cd5a8de1c97bfc8ee0f2fa54444c"
server_key_iv: "1fb8eea92fe780082c47df323996a46d"
server_key_version: "encryptionkey@0"
I20260929 02:06:38.574294 18489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:38.589350 18489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:38.593895 18489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:38.594192 18489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:38.595309 18489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:38.596647 18489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:38.596750 18489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:38.596905 18489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:38.596980 18489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:38.629712 18489 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:41601
I20260929 02:06:38.629756 18616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:41601 every 8 connection(s)
I20260929 02:06:38.630985 18489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:38.639612 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 18489
I20260929 02:06:38.639784 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:38.640352 11995 external_mini_cluster.cc:1428] Setting key b08ce770a7cbe351d6a4cad8d07e6e66
Sep 29 02:06:38 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647598, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:38.650929 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:38.633353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:40577 (local address 127.11.182.254:44915)
0929 02:06:38.633513 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:38.633518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:38.634697 (+ 1179us) server_negotiation.cc:407] Connection header received
0929 02:06:38.636438 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:38.636442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:38.636523 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:38.636609 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:38.638025 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:38.638746 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:38.641030 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:38.641669 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:38.641796 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:38.645498 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:38.645509 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:38.645512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:38.645542 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:38.648072 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:38.648738 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:38.648749 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:38.648751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:38.648872 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:38.649332 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:38.649336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:38.649338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:38.649575 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:38.649719 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:38.650366 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 02:06:38.650571 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:06:38.651750 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:38.633840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.195:40577)
0929 02:06:38.634530 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:38.634572 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:38.636199 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:38.636815 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:38.636823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:38.637297 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:38.637871 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:38.637887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:38.638952 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:38.638956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:38.640879 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:38.640890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:38.641921 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:38.641930 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:38.642228 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:38.642985 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:38.643006 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:38.645304 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:38.648210 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:38.648218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:38.648233 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:38.648612 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:38.649012 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:38.649017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:38.649021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:38.649202 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:38.649707 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:38.649714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:38.650018 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 02:06:38.650253 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 02:06:38.650719 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":258,"threads_started":1}
I20260929 02:06:38.653013 18617 heartbeater.cc:347] Connected to a master server at 127.11.182.254:44915
I20260929 02:06:38.653314 18617 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:38.654136 18617 heartbeater.cc:511] Master 127.11.182.254:44915 requested a full tablet report, sending...
I20260929 02:06:38.655673 18160 ts_manager.cc:194] Registered new tserver with Master: b81b0df2be8b4d5cb7b042ee400726d2 (127.11.182.195:41601)
I20260929 02:06:38.656500 18160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:40577
I20260929 02:06:38.667862 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:06:38.676210 18350 heartbeater.cc:503] Master 127.11.182.254:44915 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.11.182.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.253.keytab.
Entry for principal oryx/127.11.182.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.253.keytab.
I20260929 02:06:38.714213 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.253:39003
--webserver_interface=127.11.182.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.253.keytab
--principal=oryx/127.11.182.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:44915,127.11.182.253:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:39.031888 18622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:39.032166 18622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:39.032356 18622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:39.039400 18622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:39.039501 18622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:39.039615 18622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:39.039712 18622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:39.039796 18622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:39.049968 18622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:44915,127.11.182.253:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.253.keytab
--principal=oryx/127.11.182.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.253:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.182.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
W20260929 02:06:39.050343 18622 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.182.254:44915,127.11.182.253:39003'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 02:06:39.051462 18622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:39.052745 18622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:39.064316 18627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:39.064356 18630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:39.064376 18628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:39.065025 18622 server_base.cc:1034] running on GCE node
I20260929 02:06:39.065747 18622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:39.067106 18622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:39.068338 18622 hybrid_clock.cc:648] HybridClock initialized: now 1790647599068280 us; error 38 us; skew 500 ppm
I20260929 02:06:39.068661 18622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647599, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:39.071919 18622 init.cc:377] Logged in from keytab as oryx/127.11.182.253@KRBTEST.COM (short username oryx)
I20260929 02:06:39.073302 18622 webserver.cc:460] Webserver started at http://127.11.182.253:43127/ using document root <none> and password file <none>
I20260929 02:06:39.074013 18622 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:39.074097 18622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:39.074389 18622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:39.077422 18622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data/instance:
uuid: "fbda9dd051364eadbc87bc57ccd4289a"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "34b59ef31e304552a7457dddce4c9638"
server_key_iv: "4d72de23f143c0e264b1e7207e5642d8"
server_key_version: "encryptionkey@0"
I20260929 02:06:39.078223 18622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal/instance:
uuid: "fbda9dd051364eadbc87bc57ccd4289a"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "34b59ef31e304552a7457dddce4c9638"
server_key_iv: "4d72de23f143c0e264b1e7207e5642d8"
server_key_version: "encryptionkey@0"
I20260929 02:06:39.083343 18622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 02:06:39.086547 18637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:39.087873 18622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:39.088043 18622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal
uuid: "fbda9dd051364eadbc87bc57ccd4289a"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "34b59ef31e304552a7457dddce4c9638"
server_key_iv: "4d72de23f143c0e264b1e7207e5642d8"
server_key_version: "encryptionkey@0"
I20260929 02:06:39.088239 18622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:39.101195 18622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:39.105542 18622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:39.105810 18622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:39.124258 18622 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.253:39003
I20260929 02:06:39.124281 18688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.253:39003 every 8 connection(s)
I20260929 02:06:39.125557 18622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
I20260929 02:06:39.129110 18689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:39.132948 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 18622
I20260929 02:06:39.136691 18689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.253@KRBTEST.COM
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647599, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.253@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:39.155694 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.141028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.11.182.254:44915)
0929 02:06:39.141181 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.141185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.141858 (+ 673us) server_negotiation.cc:407] Connection header received
0929 02:06:39.142058 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.142063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.142133 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.142234 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.143534 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.144378 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.146090 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.146507 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.146599 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.150353 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.150358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.150361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.150390 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.152915 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.153600 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.153612 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.153615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.153736 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.154128 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.154133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.154136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.154437 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.154587 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.155130 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.155343 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:06:39.156549 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.135083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.11.182.253:39003)
0929 02:06:39.135991 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.136009 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.136371 (+ 362us) server_negotiation.cc:407] Connection header received
0929 02:06:39.137950 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.137970 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.138350 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.138691 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.140837 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.142592 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.144093 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.144923 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.145044 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.149709 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.149716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.149726 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.149759 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.152726 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.153342 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.153348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.153350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.153500 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.153886 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.153892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.153901 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.154344 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.154642 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.154999 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.155412 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":355,"threads_started":1}
I20260929 02:06:39.156592 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.141109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56400)
0929 02:06:39.141721 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.141755 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.141927 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.142292 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.142298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.143054 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.143379 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.143391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.144511 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.144515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.145935 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.145950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.146733 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.146742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.146968 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.147535 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.147562 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.150161 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.153042 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.153054 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.153064 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.153475 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.153845 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.153850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.153853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.154011 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.154566 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.154574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.154752 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.155053 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.155578 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":143,"thread_start_us":286,"threads_started":1}
I20260929 02:06:39.158347 18689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:06:39.161592 18484 heartbeater.cc:503] Master 127.11.182.254:44915 was elected leader, sending a full tablet report...
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647599, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.253@KRBTEST.COM for oryx/127.11.182.253@KRBTEST.COM
I20260929 02:06:39.169766 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.158957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.0.0.1:40366)
0929 02:06:39.159209 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.159229 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.159338 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.159678 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.159683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.160015 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.160135 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.160142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.161040 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.161043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.162085 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.162095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.162753 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.162762 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.163040 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.163570 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.163586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.165373 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.167882 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.167886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.167888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.168228 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.168522 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.168525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.168527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.168650 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.169098 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.169101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.169199 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.169300 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.169477 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":36}
I20260929 02:06:39.169845 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.159152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.11.182.253:39003)
0929 02:06:39.159307 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.159330 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.159351 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:06:39.159439 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.159447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.159515 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.159604 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.160256 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.160926 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.162204 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.162614 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.162736 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.165522 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.165526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.165529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.165554 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.167729 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.168324 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.168327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.168330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.168433 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.168732 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.168735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.168737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.168990 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.169093 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.169440 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.169595 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:06:39.174129 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: Bootstrap starting.
I20260929 02:06:39.177189 18689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:39.178200 18689 log.cc:824] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:39.180832 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: No bootstrap required, opened a new log
I20260929 02:06:39.187438 18689 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } }
I20260929 02:06:39.187711 18689 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:39.187781 18689 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbda9dd051364eadbc87bc57ccd4289a, State: Initialized, Role: FOLLOWER
I20260929 02:06:39.188244 18689 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } }
I20260929 02:06:39.189013 18695 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } } }
I20260929 02:06:39.189321 18695 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:06:39.189940 18689 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:06:39.199275 18706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:06:39.199430 18706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:06:39.201354 18622 master_runner.cc:428] Detected that this master fbda9dd051364eadbc87bc57ccd4289a is joining an existing cluster
I20260929 02:06:39.201436 18622 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.182.253:39003
I20260929 02:06:39.221490 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.205928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.11.182.253:39003)
0929 02:06:39.206112 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.206118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.206664 (+ 546us) server_negotiation.cc:407] Connection header received
0929 02:06:39.206841 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.206865 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.206946 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.207044 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.208288 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.209137 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.210913 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.211553 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.211655 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.214491 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.214499 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.214507 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.214545 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.218563 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.219450 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.219457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.219463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.219589 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.219986 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.219993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.219999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.220436 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.220568 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.221086 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.221230 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":29056}
I20260929 02:06:39.221943 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.204967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.11.182.254:44915)
0929 02:06:39.205166 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.205174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.205774 (+ 600us) server_negotiation.cc:407] Connection header received
0929 02:06:39.206050 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.206058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.206157 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.206240 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.207726 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.208639 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.210780 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.211448 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.211548 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.214239 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.214246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.214251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.214289 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.217724 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.218752 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.218758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.218762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.218989 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.219481 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.219486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.219491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.219947 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.220085 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.220934 (+ 849us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.221187 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:06:39.222234 18713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.205845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.0.0.1:40372)
0929 02:06:39.206553 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.206572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.206696 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.207092 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.207097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.207781 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.208118 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.208130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.209312 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.209317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.210723 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.210734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.211965 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.211973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.212165 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.212858 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.212885 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.214315 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.218697 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.218704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.218710 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.219327 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.219707 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.219714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.219720 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.219884 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.220644 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.220651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.220815 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.221674 (+ 859us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.221885 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":49,"thread_start_us":398,"threads_started":1}
I20260929 02:06:39.223372 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.205031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56408)
0929 02:06:39.205686 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.205707 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.205873 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.206477 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.206480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.207257 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.207590 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.207599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.208772 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.208778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.210601 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.210617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.211839 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.211847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.212021 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.212660 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.212683 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.214022 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.217920 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.217926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.217930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.218615 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.219158 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.219163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.219167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.219345 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.220195 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.220206 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.220468 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.222449 (+ 1981us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.222812 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":31,"spinlock_wait_cycles":25984,"thread_start_us":367,"threads_started":1}
WARNING: no policy specified for oryx/127.11.182.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.252@KRBTEST.COM" created.
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.254@KRBTEST.COM for oryx/127.11.182.253@KRBTEST.COM
I20260929 02:06:39.250288 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.230422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.11.182.253:39003)
0929 02:06:39.230604 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.230613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.235667 (+ 5054us) server_negotiation.cc:407] Connection header received
0929 02:06:39.236061 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.236067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.236194 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.236306 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.237922 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.238693 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.240174 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.240647 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.240809 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.243901 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.243906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.243909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.243938 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.247127 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.248009 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.248016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.248020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.248189 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.248634 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.248639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.248643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.248980 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.249128 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.249700 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.249914 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:06:39.250458 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.230390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.0.0.1:40388)
0929 02:06:39.235518 (+ 5128us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.235569 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.235829 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.236556 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.236565 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.237444 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.237744 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.237756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.238840 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.238869 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.240030 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.240042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.240837 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.240842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.241033 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.241653 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.241670 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.243718 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.247257 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.247269 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.247278 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.247850 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.248316 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.248324 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.248330 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.248497 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.249122 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.249133 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.249353 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.249568 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.249922 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4836,"thread_start_us":4712,"threads_started":1}
I20260929 02:06:39.252698 18160 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.182.253:39003: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a"
member_type: NON_VOTER
last_known_addr {
host: "127.11.182.253"
port: 39003
}
attrs {
promote: true
}
}
dest_uuid: "6c40684958ea42ad8974f33559cbcfe8"
cas_config_opid_index: -1
I20260929 02:06:39.253925 18160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } }
I20260929 02:06:39.261814 18664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:39.261392 18719 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fbda9dd051364eadbc87bc57ccd4289a (127.11.182.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } } }
I20260929 02:06:39.265727 18664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Refusing update from remote peer 6c40684958ea42ad8974f33559cbcfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 02:06:39.266008 18719 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.182.253:39003
I20260929 02:06:39.266623 18622 master.cc:537] Master@127.11.182.253:39003 shutting down...
I20260929 02:06:39.266537 18718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } } }
I20260929 02:06:39.267354 18718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:39.267725 18719 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:06:39.271204 18146 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 02:06:39.273432 18146 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 -> Peer fbda9dd051364eadbc87bc57ccd4289a (127.11.182.253:39003): Couldn't send request to peer fbda9dd051364eadbc87bc57ccd4289a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.252.keytab.
Entry for principal oryx/127.11.182.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.252.keytab.
I20260929 02:06:39.275368 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.252:41129
--webserver_interface=127.11.182.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.252.keytab
--principal=oryx/127.11.182.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:44915,127.11.182.253:39003,127.11.182.252:41129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:06:39.279395 18622 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:39.279866 18622 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:39.280074 18622 tablet_replica.cc:330] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: stopping tablet replica
I20260929 02:06:39.285996 18622 master.cc:552] Master@127.11.182.253:39003 shutdown complete.
I20260929 02:06:39.286094 18622 master_runner.cc:305] Clearing existing system tablet
I20260929 02:06:39.286535 18622 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:39.286628 18622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:39.289846 18622 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:06:39.295305 18724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:39.295693 18622 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:39.296002 18622 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.005s
I20260929 02:06:39.296114 18622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal
uuid: "fbda9dd051364eadbc87bc57ccd4289a"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "34b59ef31e304552a7457dddce4c9638"
server_key_iv: "4d72de23f143c0e264b1e7207e5642d8"
server_key_version: "encryptionkey@0"
I20260929 02:06:39.301991 18622 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:39.308759 18622 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:06:39.308923 18622 log.cc:1196] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 02:06:39.309322 18622 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: Deleting consensus metadata
I20260929 02:06:39.309599 18622 master_runner.cc:315] Copying system tablet from 127.11.182.254:44915
I20260929 02:06:39.312105 18622 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: tablet copy: Beginning tablet copy session from remote peer at address 127.11.182.254:44915
I20260929 02:06:39.327075 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.312917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.11.182.254:44915)
0929 02:06:39.313097 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.313105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.313508 (+ 403us) server_negotiation.cc:407] Connection header received
0929 02:06:39.313743 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.313750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.313845 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.313954 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.314962 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.316089 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.317741 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.318394 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.318524 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.320729 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.320738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.320743 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.320784 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.324195 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.324941 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.324947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.324951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.325089 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.325515 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.325520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.325524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.325879 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.326039 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.326451 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.326675 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:06:39.327183 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.312904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56412)
0929 02:06:39.313397 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.313435 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.313577 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.313992 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.313996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.314505 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.314793 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.314804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.316246 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.316252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.317586 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.317601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.318560 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.318565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.318738 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.319495 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.319517 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.320502 (+ 985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.324331 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.324336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.324339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.324824 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.325206 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.325210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.325214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.325386 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.326003 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.326007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.326193 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.326344 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.326624 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":267,"threads_started":1}
I20260929 02:06:39.328539 18181 tablet_copy_service.cc:140] P 6c40684958ea42ad8974f33559cbcfe8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fbda9dd051364eadbc87bc57ccd4289a ({username='oryx', principal='oryx/127.11.182.253@KRBTEST.COM'} at 127.0.0.1:56412)
I20260929 02:06:39.328855 18181 tablet_copy_service.cc:161] P 6c40684958ea42ad8974f33559cbcfe8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fbda9dd051364eadbc87bc57ccd4289a at {username='oryx', principal='oryx/127.11.182.253@KRBTEST.COM'} at 127.0.0.1:56412: session id = fbda9dd051364eadbc87bc57ccd4289a-00000000000000000000000000000000
I20260929 02:06:39.333408 18181 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:06:39.335705 18622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:06:39.343320 18622 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: tablet copy: Starting download of 0 data blocks...
I20260929 02:06:39.343624 18622 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: tablet copy: Starting download of 1 WAL segments...
I20260929 02:06:39.346240 18622 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:06:39.350898 18181 tablet_copy_service.cc:342] P 6c40684958ea42ad8974f33559cbcfe8: Request end of tablet copy session fbda9dd051364eadbc87bc57ccd4289a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.182.253@KRBTEST.COM'} at 127.0.0.1:56412
I20260929 02:06:39.351121 18181 tablet_copy_service.cc:433] P 6c40684958ea42ad8974f33559cbcfe8: ending tablet copy session fbda9dd051364eadbc87bc57ccd4289a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fbda9dd051364eadbc87bc57ccd4289a
I20260929 02:06:39.363893 18622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:39.366783 18731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:39.367254 18732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:39.368391 18734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:39.371532 18622 server_base.cc:1034] running on GCE node
I20260929 02:06:39.371908 18622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:39.372576 18622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:39.373792 18622 hybrid_clock.cc:648] HybridClock initialized: now 1790647599373730 us; error 35 us; skew 500 ppm
I20260929 02:06:39.374008 18622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647599, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:39.376156 18622 init.cc:377] Logged in from keytab as oryx/127.11.182.253@KRBTEST.COM (short username oryx)
I20260929 02:06:39.383463 18622 webserver.cc:460] Webserver started at http://127.11.182.253:46681/ using document root <none> and password file <none>
I20260929 02:06:39.383823 18622 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:39.383905 18622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:39.387008 18622 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 02:06:39.388861 18741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:39.389372 18622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260929 02:06:39.389549 18622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal
uuid: "fbda9dd051364eadbc87bc57ccd4289a"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "34b59ef31e304552a7457dddce4c9638"
server_key_iv: "4d72de23f143c0e264b1e7207e5642d8"
server_key_version: "encryptionkey@0"
I20260929 02:06:39.389780 18622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:39.408818 18622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:39.412999 18622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:39.441391 18622 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.253:39003
I20260929 02:06:39.441430 18792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.253:39003 every 8 connection(s)
I20260929 02:06:39.442523 18622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
I20260929 02:06:39.449165 18793 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:06:39.453089 18793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: Bootstrap starting.
I20260929 02:06:39.470798 18793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:06:39.471414 18793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: Bootstrap complete.
I20260929 02:06:39.472523 18793 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } }
I20260929 02:06:39.472790 18793 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fbda9dd051364eadbc87bc57ccd4289a, State: Initialized, Role: LEARNER
I20260929 02:06:39.473014 18793 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } }
I20260929 02:06:39.474037 18793 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:39.473838 18796 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } } }
I20260929 02:06:39.474278 18796 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: This master's current role is: LEARNER
I20260929 02:06:39.483191 18807 catalog_manager.cc:1259] Loaded cluster ID: eea07e915f3a4c728db042c3d44580c3
I20260929 02:06:39.483305 18807 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: loading cluster ID for follower catalog manager: success
I20260929 02:06:39.485639 18807 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: acquiring CA information for follower catalog manager: success
I20260929 02:06:39.487550 18807 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:06:39.658962 18617 heartbeater.cc:503] Master 127.11.182.254:44915 was elected leader, sending a full tablet report...
I20260929 02:06:39.660813 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.653108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.0.0.1:40404)
0929 02:06:39.653387 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.653408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.653551 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.654616 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.654622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.654660 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:39.654910 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.654920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.656689 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.656697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.657949 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.657962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.660012 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.660018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.660251 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.660316 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.660388 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.660520 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 02:06:39.661891 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.653302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.11.182.253:39003)
0929 02:06:39.654267 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.654271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.654290 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:06:39.654352 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.654355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.654422 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.654567 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:39.655052 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.656554 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.658156 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.659795 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.661241 (+ 1446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.661446 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.661502 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.661627 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":692,"threads_started":1}
I20260929 02:06:39.663411 18768 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 02:06:39.664072 18796 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c40684958ea42ad8974f33559cbcfe8. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } } }
I20260929 02:06:39.664356 18796 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: This master's current role is: LEARNER
W20260929 02:06:39.713189 18722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:39.713542 18722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:39.713693 18722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:39.720968 18722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:39.721151 18722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:39.721252 18722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:39.721354 18722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:39.721426 18722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:39.731254 18722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:44915,127.11.182.253:39003,127.11.182.252:41129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.252.keytab
--principal=oryx/127.11.182.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.252:41129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.182.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:39.732684 18722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:39.733989 18722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:39.749368 18816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:39.749368 18814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:39.750118 18813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:39.750229 18722 server_base.cc:1034] running on GCE node
I20260929 02:06:39.751228 18722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:39.752684 18722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:39.753878 18722 hybrid_clock.cc:648] HybridClock initialized: now 1790647599753833 us; error 39 us; skew 500 ppm
I20260929 02:06:39.754227 18722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647599, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:39.757611 18722 init.cc:377] Logged in from keytab as oryx/127.11.182.252@KRBTEST.COM (short username oryx)
I20260929 02:06:39.759213 18722 webserver.cc:460] Webserver started at http://127.11.182.252:44601/ using document root <none> and password file <none>
I20260929 02:06:39.760115 18722 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:39.760264 18722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:39.760613 18722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:39.763701 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data/instance:
uuid: "bafae58e078f44b9bc3fc977f92c4c43"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "f1cadd4380190be8cfd64b6795f863f3"
server_key_iv: "5d28d8b0ba6aaf2321d9d37cebbdbc4c"
server_key_version: "encryptionkey@0"
I20260929 02:06:39.764456 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal/instance:
uuid: "bafae58e078f44b9bc3fc977f92c4c43"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "f1cadd4380190be8cfd64b6795f863f3"
server_key_iv: "5d28d8b0ba6aaf2321d9d37cebbdbc4c"
server_key_version: "encryptionkey@0"
I20260929 02:06:39.769470 18722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:06:39.772804 18823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:39.774369 18722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:39.774610 18722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal
uuid: "bafae58e078f44b9bc3fc977f92c4c43"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "f1cadd4380190be8cfd64b6795f863f3"
server_key_iv: "5d28d8b0ba6aaf2321d9d37cebbdbc4c"
server_key_version: "encryptionkey@0"
I20260929 02:06:39.774814 18722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:39.792284 18722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:39.796321 18722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:39.796608 18722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:39.816223 18722 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.252:41129
I20260929 02:06:39.816252 18874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.252:41129 every 8 connection(s)
I20260929 02:06:39.817557 18722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
I20260929 02:06:39.821151 18875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:39.824400 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 18722
I20260929 02:06:39.828320 18875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.252@KRBTEST.COM
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647599, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.252@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:39.846653 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.831418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.11.182.254:44915)
0929 02:06:39.832665 (+ 1247us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.832669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.832684 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:06:39.832729 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.832732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.832813 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.832895 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.834187 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.835292 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.837024 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.837411 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.837540 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.841217 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.841225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.841232 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.841284 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.843856 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.844593 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.844601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.844608 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.844745 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.845159 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.845165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.845169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.845429 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.845586 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.846094 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.846297 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":1059,"threads_started":1}
I20260929 02:06:39.847718 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.826454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.11.182.252:41129)
0929 02:06:39.827249 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.827260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.827299 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:06:39.828988 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.829010 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.829403 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.829789 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.831236 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.833571 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.835237 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.836230 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.836407 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.840710 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.840717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.840729 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.840761 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.843310 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.844552 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.844569 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.844572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.844728 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.845161 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.845172 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.845174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.845662 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.846005 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.846321 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.846646 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":612,"thread_start_us":410,"threads_started":1}
I20260929 02:06:39.847734 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.831215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56424)
0929 02:06:39.832001 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.832055 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.832227 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.833003 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.833011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.833810 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.834041 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.834049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.835422 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.835427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.836881 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.836891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.837619 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.837625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.837777 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.838355 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.838376 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.841036 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.843992 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.844004 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.844021 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.844405 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.844890 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.844896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.844904 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.845047 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.845557 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.845564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.845745 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.846013 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.846500 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":262,"threads_started":1}
I20260929 02:06:39.849997 18875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647599, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.252@KRBTEST.COM for oryx/127.11.182.253@KRBTEST.COM
I20260929 02:06:39.861764 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.850614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.0.0.1:40418)
0929 02:06:39.850796 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.850810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.850935 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.851291 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.851295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.851618 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.851738 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.851745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.852704 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.852712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.853602 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.853612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.854294 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.854299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.854520 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.855136 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.855151 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.856968 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.859808 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.859816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.859823 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.860237 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.860547 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.860550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.860552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.860671 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.861123 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.861127 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.861235 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.861350 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.861545 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:06:39.861830 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.850820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.11.182.253:39003)
0929 02:06:39.851039 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.851045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.851061 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:06:39.851118 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.851122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.851174 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.851282 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.851883 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.852582 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.853743 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.854150 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.854285 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.857135 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.857140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.857143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.857174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.859689 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.860340 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.860344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.860346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.860458 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.860756 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.860759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.860761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.861029 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.861133 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.861430 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.861594 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 02:06:39.862644 18875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647599, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.252@KRBTEST.COM for oryx/127.11.182.252@KRBTEST.COM
I20260929 02:06:39.875007 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.863463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:41129 (local address 127.0.0.1:57184)
0929 02:06:39.863684 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.863709 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.863832 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.864101 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.864105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.864457 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.864579 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.864586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.865477 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.865485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.866693 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.866704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.867397 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.867402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.867618 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.868106 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.868121 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.869939 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.872570 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.872578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.872581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.872922 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.873276 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.873282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.873285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.873428 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.873969 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.873974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.874121 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.874265 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.874549 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 02:06:39.875097 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.863503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.11.182.252:41129)
0929 02:06:39.863664 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.863669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.863785 (+ 116us) server_negotiation.cc:407] Connection header received
0929 02:06:39.863934 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.863940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.863994 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.864087 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.864703 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.865362 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.866870 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.867262 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.867394 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.870114 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.870119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.870122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.870147 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.872440 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.873030 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.873038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.873042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.873157 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.873530 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.873537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.873540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.873855 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.874026 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.874328 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.874674 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":50}
I20260929 02:06:39.880254 18875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Bootstrap starting.
I20260929 02:06:39.883399 18875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:39.884356 18875 log.cc:824] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:39.887060 18875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: No bootstrap required, opened a new log
I20260929 02:06:39.894708 18875 raft_consensus.cc:348] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } }
I20260929 02:06:39.895085 18875 raft_consensus.cc:374] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:39.895177 18875 raft_consensus.cc:729] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bafae58e078f44b9bc3fc977f92c4c43, State: Initialized, Role: FOLLOWER
I20260929 02:06:39.895759 18875 consensus_queue.cc:260] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } }
I20260929 02:06:39.896618 18882 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } } }
I20260929 02:06:39.896930 18882 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:06:39.897653 18875 sys_catalog.cc:564] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:06:39.907539 18893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:06:39.907697 18893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:06:39.912436 18722 master_runner.cc:428] Detected that this master bafae58e078f44b9bc3fc977f92c4c43 is joining an existing cluster
I20260929 02:06:39.912571 18722 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.182.252:41129
I20260929 02:06:39.934123 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.916855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.11.182.254:44915)
0929 02:06:39.917049 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.917055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.918076 (+ 1021us) server_negotiation.cc:407] Connection header received
0929 02:06:39.918353 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.918359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.918445 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.918517 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.920039 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.921165 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.923276 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.924363 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.924475 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.927208 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.927216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.927221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.927261 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.930776 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.931644 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.931650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.931654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.931821 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.932299 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.932304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.932308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.932651 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.932797 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.933491 (+ 694us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.933723 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:06:39.935277 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.917214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56436)
0929 02:06:39.917967 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.917987 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.918191 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.918730 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.918735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.919468 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.919856 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.919870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.921356 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.921362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.923101 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.923122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.924775 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.924782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.924970 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.925745 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.925782 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.926978 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.930973 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.930977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.930981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.931499 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.931974 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.931978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.931982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.932168 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.932880 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.932885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.933140 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.934410 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.934744 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":457,"threads_started":1}
I20260929 02:06:39.947948 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.916871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.11.182.252:41129)
0929 02:06:39.917118 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.917130 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.928486 (+ 11356us) server_negotiation.cc:407] Connection header received
0929 02:06:39.928936 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.928944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.929025 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.929108 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.930907 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.932456 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.935641 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.936483 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.936554 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.939931 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.939939 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.939947 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.939985 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.944300 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.945215 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.945223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.945229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.945386 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.945885 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.945893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.945898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.946334 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.946519 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.947136 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.947413 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:06:39.949837 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.917410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:41129 (local address 127.0.0.1:57198)
0929 02:06:39.928363 (+ 10953us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.928390 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.928785 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.929354 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.929361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.930259 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.930697 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.930713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.932720 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.932728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.934812 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.934834 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.936880 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.936891 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.937060 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.937861 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.937886 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.939715 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.944476 (+ 4761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.944483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.944490 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.945050 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.945540 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.945547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.945553 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.945737 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.946600 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.946607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.946814 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.949079 (+ 2265us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.949367 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10734,"mutex_wait_us":28,"thread_start_us":17616,"threads_started":1}
I20260929 02:06:39.952440 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.918653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.11.182.253:39003)
0929 02:06:39.918909 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.918917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.935502 (+ 16585us) server_negotiation.cc:407] Connection header received
0929 02:06:39.937007 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.937013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.937100 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.937176 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.938667 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.939803 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.942016 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.942661 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.942766 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.945687 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.945693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.945700 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.945740 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.949059 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.949879 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.949884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.949888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.950033 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.950475 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.950480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.950483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.950789 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.950926 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.951888 (+ 962us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.952104 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:06:39.953387 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.918248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.0.0.1:40422)
0929 02:06:39.935381 (+ 17133us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.935403 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.936836 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.937411 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.937418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.938198 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.938530 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.938544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.939978 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.939984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.941844 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.941858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.943078 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.943085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.943798 (+ 713us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.944381 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.944403 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.945491 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.949226 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.949230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.949234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.949721 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.950173 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.950178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.950181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.950338 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.951048 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.951053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.951595 (+ 542us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.952671 (+ 1076us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.952969 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16939,"thread_start_us":10244,"threads_started":1}
I20260929 02:06:39.953544 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 18222
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.254@KRBTEST.COM for oryx/127.11.182.252@KRBTEST.COM
I20260929 02:06:39.965085 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.942135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:41129 (local address 127.0.0.1:57212)
0929 02:06:39.942532 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:39.942552 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:39.942655 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:39.948973 (+ 6318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:39.948978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:39.949443 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:39.949637 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.949648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.950979 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.950984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.952178 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:39.952193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.953251 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.953256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:39.953425 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.954138 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:39.954164 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:39.957534 (+ 3370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:39.962003 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.962010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.962016 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.962484 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.962965 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:39.962970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:39.962973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:39.963126 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:39.964087 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:39.964092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:39.964259 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 02:06:39.964415 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 02:06:39.964681 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260929 02:06:39.965770 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.943002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.11.182.252:41129)
0929 02:06:39.948516 (+ 5514us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.948524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.948547 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:06:39.948655 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.948663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.948833 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.948939 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.949826 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.950808 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.952359 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.953085 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.953709 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.957742 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.957749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.957785 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.957826 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.961843 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.962641 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.962646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.962650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.962810 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.963296 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.963301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.963304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.963958 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.964103 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.964739 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.965061 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5383,"thread_start_us":20591,"threads_started":1}
I20260929 02:06:39.967576 18160 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.182.252:41129: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43"
member_type: NON_VOTER
last_known_addr {
host: "127.11.182.252"
port: 41129
}
attrs {
promote: true
}
}
dest_uuid: "6c40684958ea42ad8974f33559cbcfe8"
cas_config_opid_index: 5
I20260929 02:06:39.968844 18160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } }
I20260929 02:06:39.970916 18904 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bafae58e078f44b9bc3fc977f92c4c43 (127.11.182.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:39.972402 18768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 LEARNER]: Refusing update from remote peer 6c40684958ea42ad8974f33559cbcfe8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:06:39.973176 18718 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 02:06:39.978024 18850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 02:06:39 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.193@KRBTEST.COM
I20260929 02:06:39.982782 18850 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 FOLLOWER]: Refusing update from remote peer 6c40684958ea42ad8974f33559cbcfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:06:39.983695 18718 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:39.984256 18904 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.182.252:41129
I20260929 02:06:39.984859 18722 master.cc:537] Master@127.11.182.252:41129 shutting down...
I20260929 02:06:39.989511 18718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:39.989753 18718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:39.989511 18768 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bafae58e078f44b9bc3fc977f92c4c43 (127.11.182.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:39.991680 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:39.956513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.11.182.193:35255)
0929 02:06:39.969034 (+ 12521us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:39.969044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:39.969076 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:06:39.969143 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:39.969151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:39.969441 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:39.969617 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:39.971288 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.973075 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.975176 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:39.976050 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:39.982537 (+ 6487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:39.982825 (+ 288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:39.982838 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:39.982887 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:39.982931 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:39.987358 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.988223 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.988232 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.988236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.988390 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:39.988890 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:39.988894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:39.988898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:39.989353 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:39.990881 (+ 1528us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:39.991043 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 02:06:39.991295 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12333,"thread_start_us":13313,"threads_started":1}
I20260929 02:06:39.993310 18345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:39.996742 18796 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: NON_VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: true } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:39.997030 18796 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: This master's current role is: LEARNER
I20260929 02:06:39.998230 18904 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: attempting to promote NON_VOTER fbda9dd051364eadbc87bc57ccd4289a to VOTER
I20260929 02:06:39.999202 18904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } }
I20260929 02:06:40.003614 18768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 LEARNER]: Refusing update from remote peer 6c40684958ea42ad8974f33559cbcfe8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 02:06:40.004225 18718 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 02:06:40.007220 18768 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fbda9dd051364eadbc87bc57ccd4289a (127.11.182.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:40.018517 18796 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:40.018877 18796 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:06:40.026062 18722 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:40.026582 18722 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:40.026806 18722 tablet_replica.cc:330] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: stopping tablet replica
I20260929 02:06:40.027551 18718 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fbda9dd051364eadbc87bc57ccd4289a (127.11.182.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
W20260929 02:06:40.028614 18145 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 -> Peer bafae58e078f44b9bc3fc977f92c4c43 (127.11.182.252:41129): Couldn't send request to peer bafae58e078f44b9bc3fc977f92c4c43. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:06:40.032402 18905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:40.032802 18905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:40.033001 18905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:40.033217 18905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:40.042527 18722 master.cc:552] Master@127.11.182.252:41129 shutdown complete.
I20260929 02:06:40.042620 18722 master_runner.cc:305] Clearing existing system tablet
I20260929 02:06:40.043072 18722 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:40.043187 18722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:40.045971 18722 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:06:40.047554 18914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:40.047844 18722 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:40.048108 18722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 02:06:40.048218 18722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal
uuid: "bafae58e078f44b9bc3fc977f92c4c43"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "f1cadd4380190be8cfd64b6795f863f3"
server_key_iv: "5d28d8b0ba6aaf2321d9d37cebbdbc4c"
server_key_version: "encryptionkey@0"
I20260929 02:06:40.054721 18722 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:40.062474 18722 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:06:40.062650 18722 log.cc:1196] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 02:06:40.063202 18722 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Deleting consensus metadata
I20260929 02:06:40.063455 18722 master_runner.cc:315] Copying system tablet from 127.11.182.254:44915
I20260929 02:06:40.075285 18722 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: tablet copy: Beginning tablet copy session from remote peer at address 127.11.182.254:44915
I20260929 02:06:40.098071 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.076156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56440)
0929 02:06:40.083602 (+ 7446us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:40.083623 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:40.083789 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:40.084186 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:40.084191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:40.084721 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:40.084992 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.085004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.086454 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.086467 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.088024 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.088040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.089010 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.089017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.089195 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.089955 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:40.089979 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:40.091176 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:40.095152 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:40.095158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:40.095163 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:40.095689 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:40.096112 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:40.096117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:40.096122 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:40.096301 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:40.096880 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:40.096885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:40.097098 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 02:06:40.097248 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 02:06:40.097608 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7292,"thread_start_us":7154,"threads_started":1}
I20260929 02:06:40.098775 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.076156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.11.182.254:44915)
0929 02:06:40.076390 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:40.076397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:40.083743 (+ 7346us) server_negotiation.cc:407] Connection header received
0929 02:06:40.083929 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:40.083934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:40.084016 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:40.084131 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:40.085165 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.086299 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.088177 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.088839 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.088959 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.091403 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:40.091410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:40.091415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:40.091454 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:40.094981 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.095835 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.095841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.095845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.095994 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.096419 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.096424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.096428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.096761 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:40.096895 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:40.097395 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 02:06:40.098116 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 02:06:40.099591 18181 tablet_copy_service.cc:140] P 6c40684958ea42ad8974f33559cbcfe8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bafae58e078f44b9bc3fc977f92c4c43 ({username='oryx', principal='oryx/127.11.182.252@KRBTEST.COM'} at 127.0.0.1:56440)
I20260929 02:06:40.099767 18181 tablet_copy_service.cc:161] P 6c40684958ea42ad8974f33559cbcfe8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bafae58e078f44b9bc3fc977f92c4c43 at {username='oryx', principal='oryx/127.11.182.252@KRBTEST.COM'} at 127.0.0.1:56440: session id = bafae58e078f44b9bc3fc977f92c4c43-00000000000000000000000000000000
I20260929 02:06:40.103428 18181 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:06:40.105268 18722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:06:40.112038 18722 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: tablet copy: Starting download of 0 data blocks...
I20260929 02:06:40.112339 18722 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: tablet copy: Starting download of 1 WAL segments...
I20260929 02:06:40.115258 18722 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:06:40.119530 18181 tablet_copy_service.cc:342] P 6c40684958ea42ad8974f33559cbcfe8: Request end of tablet copy session bafae58e078f44b9bc3fc977f92c4c43-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.182.252@KRBTEST.COM'} at 127.0.0.1:56440
I20260929 02:06:40.119674 18181 tablet_copy_service.cc:433] P 6c40684958ea42ad8974f33559cbcfe8: ending tablet copy session bafae58e078f44b9bc3fc977f92c4c43-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bafae58e078f44b9bc3fc977f92c4c43
I20260929 02:06:40.129565 18722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:40.136989 18921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:40.141338 18922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:40.141582 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 18222
W20260929 02:06:40.149837 18924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:40.149957 18722 server_base.cc:1034] running on GCE node
I20260929 02:06:40.150261 18722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:40.150902 18722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:40.152103 18722 hybrid_clock.cc:648] HybridClock initialized: now 1790647600152044 us; error 51 us; skew 500 ppm
I20260929 02:06:40.152339 18722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:40 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647600, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:40.154168 18722 init.cc:377] Logged in from keytab as oryx/127.11.182.252@KRBTEST.COM (short username oryx)
I20260929 02:06:40.160295 18722 webserver.cc:460] Webserver started at http://127.11.182.252:34663/ using document root <none> and password file <none>
I20260929 02:06:40.160606 18722 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:40.160728 18722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:40.163048 18722 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:06:40.165697 18931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:40.166103 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:35255
--local_ip_for_outbound_sockets=127.11.182.193
--tserver_master_addrs=127.11.182.254:44915
--webserver_port=36249
--webserver_interface=127.11.182.193
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab
--principal=oryx/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.182.254:44915,127.11.182.253:39003
--tserver_master_addrs=127.11.182.254:44915,127.11.182.253:39003,127.11.182.252:41129 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:06:40.166606 18722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:06:40.166752 18722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal
uuid: "bafae58e078f44b9bc3fc977f92c4c43"
format_stamp: "Formatted at 2026-09-29 02:06:39 on dist-test-slave-q06p"
server_key: "f1cadd4380190be8cfd64b6795f863f3"
server_key_iv: "5d28d8b0ba6aaf2321d9d37cebbdbc4c"
server_key_version: "encryptionkey@0"
I20260929 02:06:40.166961 18722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:40.188997 18722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:40.192529 18722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:40.225121 18722 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.252:41129
I20260929 02:06:40.225142 18983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.252:41129 every 8 connection(s)
I20260929 02:06:40.226049 18722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
I20260929 02:06:40.233129 18984 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:06:40.238749 18984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Bootstrap starting.
I20260929 02:06:40.259059 18984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:06:40.259562 18984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Bootstrap complete.
I20260929 02:06:40.260494 18984 raft_consensus.cc:348] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } }
I20260929 02:06:40.260812 18984 raft_consensus.cc:729] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bafae58e078f44b9bc3fc977f92c4c43, State: Initialized, Role: LEARNER
I20260929 02:06:40.261078 18984 consensus_queue.cc:260] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } }
I20260929 02:06:40.263299 18987 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:40.263649 18984 sys_catalog.cc:564] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:40.263768 18987 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:06:40.280505 18998 catalog_manager.cc:1259] Loaded cluster ID: eea07e915f3a4c728db042c3d44580c3
I20260929 02:06:40.280735 18998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: loading cluster ID for follower catalog manager: success
I20260929 02:06:40.283134 18998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: acquiring CA information for follower catalog manager: success
I20260929 02:06:40.284793 18998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:06:40.534716 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.525435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:41129 (local address 127.0.0.1:57220)
0929 02:06:40.526251 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:40.526287 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:40.526464 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:40.527449 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:40.527455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:40.527481 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:40.527824 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.527838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.529920 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.529927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.532047 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.532063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.533211 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.533217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.534060 (+ 843us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.534095 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:06:40.534231 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 02:06:40.534449 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":250,"threads_started":1}
I20260929 02:06:40.534804 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.525565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.11.182.252:41129)
0929 02:06:40.527034 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:40.527039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:40.527054 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:06:40.527141 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:40.527146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:40.527265 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:40.527585 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:40.528051 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.529781 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.532206 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.533008 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.533206 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.533420 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:40.534265 (+ 845us) server_negotiation.cc:299] Negotiation successful
0929 02:06:40.534426 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":1147,"threads_started":1}
I20260929 02:06:40.535835 18959 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 02:06:40.536461 18987 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c40684958ea42ad8974f33559cbcfe8. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:40.536767 18987 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: This master's current role is: LEARNER
W20260929 02:06:40.564869 18934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:40.565194 18934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:40.565384 18934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:40.573489 18934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:40.573601 18934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:40.573786 18934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:40.584756 18934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab
--principal=oryx/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:35255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=36249
--tserver_master_addrs=127.11.182.254:44915,127.11.182.253:39003,127.11.182.252:41129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:40.586153 18934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:40.587550 18934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:40.601439 19006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:40.601948 19008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:40.601997 19005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:40.602321 18934 server_base.cc:1034] running on GCE node
I20260929 02:06:40.603056 18934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:40.603976 18934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:40.605201 18934 hybrid_clock.cc:648] HybridClock initialized: now 1790647600605159 us; error 48 us; skew 500 ppm
I20260929 02:06:40.605516 18934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:40 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647600, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:40.608686 18934 init.cc:377] Logged in from keytab as oryx/127.11.182.193@KRBTEST.COM (short username oryx)
I20260929 02:06:40.610062 18934 webserver.cc:460] Webserver started at http://127.11.182.193:36249/ using document root <none> and password file <none>
I20260929 02:06:40.610750 18934 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:40.610935 18934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:40.616230 18934 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 02:06:40.618906 19015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:40.620268 18934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:06:40.620445 18934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "27c9eede04a541b1b1f2dd6b37a882a1"
format_stamp: "Formatted at 2026-09-29 02:06:37 on dist-test-slave-q06p"
server_key: "5432f2d6ab1a6eb39ace7a1f1f6f7f76"
server_key_iv: "e65a66c6536e719123898e4db136dc08"
server_key_version: "encryptionkey@0"
I20260929 02:06:40.621541 18934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:40.641366 18934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:40.645160 18934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:40.645401 18934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:40.646294 18934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:40.647642 18934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:40.647728 18934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:40.647892 18934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:40.647996 18934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:40.686625 18934 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:35255
I20260929 02:06:40.686707 19127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:35255 every 8 connection(s)
I20260929 02:06:40.688058 18934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:40.693017 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 18934
I20260929 02:06:40.693408 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 18356
Sep 29 02:06:40 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647600, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.252@KRBTEST.COM
Sep 29 02:06:40 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647600, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.253@KRBTEST.COM
I20260929 02:06:40.721398 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.691786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:44741 (local address 127.11.182.252:41129)
0929 02:06:40.691988 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:40.691997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:40.694951 (+ 2954us) server_negotiation.cc:407] Connection header received
0929 02:06:40.702152 (+ 7201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:40.702162 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:40.702293 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:40.702384 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:40.704837 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.705942 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.708495 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.709141 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.709236 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.714308 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:40.714314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:40.714320 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:40.714362 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:40.717738 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.718653 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.718661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.718668 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.718829 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.719387 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.719393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.719395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.719749 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:40.719912 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:40.720853 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 02:06:40.721069 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:06:40.723160 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.691364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:44415 (local address 127.11.182.253:39003)
0929 02:06:40.703922 (+ 12558us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:40.703928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:40.705513 (+ 1585us) server_negotiation.cc:407] Connection header received
0929 02:06:40.705573 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:40.705578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:40.705659 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:40.706887 (+ 1228us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:40.707044 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.708079 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.710395 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.710975 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.711097 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.715066 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:40.715071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:40.715074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:40.715134 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:40.718609 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.719409 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.719413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.719416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.719534 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.719959 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.719964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.719967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.720275 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:40.722408 (+ 2133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:40.722567 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 02:06:40.722777 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12429,"thread_start_us":12553,"threads_started":1}
I20260929 02:06:40.723920 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.691912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:41129 (local address 127.11.182.193:44741)
0929 02:06:40.694725 (+ 2813us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:40.694790 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:40.701726 (+ 6936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:40.703100 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:40.703119 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:40.704018 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:40.704647 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.704670 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.706133 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.706140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.708321 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.708339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.709552 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.709558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.710236 (+ 678us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.711084 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:40.711114 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:40.714037 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:40.717902 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:40.717912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:40.717929 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:40.718514 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:40.719005 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:40.719011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:40.719017 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:40.719246 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:40.719996 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:40.720005 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:40.720427 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 02:06:40.721645 (+ 1218us) client_negotiation.cc:240] Negotiation successful
0929 02:06:40.722413 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2427,"thread_start_us":12927,"threads_started":1}
I20260929 02:06:40.725368 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.691912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.11.182.193:44415)
0929 02:06:40.705267 (+ 13355us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:40.705290 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:40.705411 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:40.705803 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:40.705807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:40.706412 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:40.706717 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.706728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.708239 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.708245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.710202 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.710214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.711319 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.711326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.711509 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.712061 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:40.712081 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:40.714828 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:40.718740 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:40.718748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:40.718752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:40.719227 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:40.719658 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:40.719663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:40.719667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:40.719829 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:40.720418 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:40.720422 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:40.720579 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 02:06:40.720715 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 02:06:40.721182 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13187,"mutex_wait_us":33,"spinlock_wait_cycles":72320,"thread_start_us":2317,"threads_started":1}
I20260929 02:06:40.725431 19128 heartbeater.cc:347] Connected to a master server at 127.11.182.252:41129
I20260929 02:06:40.725765 19128 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:40.726378 19130 heartbeater.cc:347] Connected to a master server at 127.11.182.253:39003
I20260929 02:06:40.726498 19130 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:40.727305 19128 heartbeater.cc:511] Master 127.11.182.252:41129 requested a full tablet report, sending...
I20260929 02:06:40.727325 19130 heartbeater.cc:511] Master 127.11.182.253:39003 requested a full tablet report, sending...
I20260929 02:06:40.729628 18758 ts_manager.cc:194] Registered new tserver with Master: 27c9eede04a541b1b1f2dd6b37a882a1 (127.11.182.193:35255)
I20260929 02:06:40.729604 18949 ts_manager.cc:194] Registered new tserver with Master: 27c9eede04a541b1b1f2dd6b37a882a1 (127.11.182.193:35255)
Sep 29 02:06:40 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647600, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
Sep 29 02:06:40 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.194@KRBTEST.COM
I20260929 02:06:40.740061 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.695482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.193:53249)
0929 02:06:40.720176 (+ 24694us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:40.720201 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:40.720375 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:40.721669 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:40.721676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:40.723789 (+ 2113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:40.724121 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.724134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.726154 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.726161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.727905 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:40.727919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.730285 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.730292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:40.730495 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.731004 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:40.731029 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:40.734026 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:40.737748 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:40.737752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:40.737756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:40.738067 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:40.738438 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:40.738441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:40.738443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:40.738568 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:40.739177 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:40.739182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:40.739358 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 02:06:40.739504 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 02:06:40.739779 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24537,"thread_start_us":24385,"threads_started":1}
I20260929 02:06:40.740310 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.695858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:53249 (local address 127.11.182.254:44915)
0929 02:06:40.719607 (+ 23749us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:40.719616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:40.720518 (+ 902us) server_negotiation.cc:407] Connection header received
0929 02:06:40.720601 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:40.720610 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:40.720711 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:40.720846 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:40.724338 (+ 3492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.726015 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.728056 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.728656 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.728789 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.734275 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:40.734285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:40.734292 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:40.734331 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:40.737625 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.738193 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.738199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.738202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.738335 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.738676 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.738684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.738689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.739070 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:40.739209 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:40.739676 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 02:06:40.739915 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23589,"thread_start_us":23764,"threads_started":1}
I20260929 02:06:40.741204 19129 heartbeater.cc:347] Connected to a master server at 127.11.182.254:44915
I20260929 02:06:40.741343 19129 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:40.741807 19129 heartbeater.cc:511] Master 127.11.182.254:44915 requested a full tablet report, sending...
I20260929 02:06:40.742508 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:40.694461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.11.182.194:34729)
0929 02:06:40.721555 (+ 27094us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:40.721568 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:40.722564 (+ 996us) server_negotiation.cc:407] Connection header received
0929 02:06:40.722644 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:40.722652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:40.722933 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:40.723096 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:40.725229 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.726995 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.728826 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:40.729607 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:40.729738 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:40.735671 (+ 5933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:40.735684 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:40.735701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:40.735745 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:40.739036 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.739847 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.739857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.739861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.740024 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:40.740701 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:40.740706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:40.740710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:40.741175 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:40.741470 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:40.741754 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 02:06:40.741995 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26868,"thread_start_us":27864,"threads_started":1}
I20260929 02:06:40.743103 18160 ts_manager.cc:194] Re-registered known tserver with Master: 27c9eede04a541b1b1f2dd6b37a882a1 (127.11.182.193:35255)
I20260929 02:06:40.743884 18479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:40.744151 18160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:53249
I20260929 02:06:40.828368 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 18356
I20260929 02:06:40.844429 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:34729
--local_ip_for_outbound_sockets=127.11.182.194
--tserver_master_addrs=127.11.182.254:44915
--webserver_port=42141
--webserver_interface=127.11.182.194
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab
--principal=oryx/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.182.254:44915,127.11.182.253:39003
--tserver_master_addrs=127.11.182.254:44915,127.11.182.253:39003,127.11.182.252:41129 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:06:41.132740 18911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: NON_VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: true } } }
I20260929 02:06:41.133185 18911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:41.134706 19141 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: attempting to promote NON_VOTER bafae58e078f44b9bc3fc977f92c4c43 to VOTER
I20260929 02:06:41.135255 19141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } }
I20260929 02:06:41.137197 18959 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 LEARNER]: Refusing update from remote peer 6c40684958ea42ad8974f33559cbcfe8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:06:41.137368 18768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Refusing update from remote peer 6c40684958ea42ad8974f33559cbcfe8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:06:41.137866 19142 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:06:41.138340 19141 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:06:41.141440 19141 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bafae58e078f44b9bc3fc977f92c4c43 (127.11.182.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:41.142228 18768 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bafae58e078f44b9bc3fc977f92c4c43 (127.11.182.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:41.145133 19141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:41.144608 18959 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bafae58e078f44b9bc3fc977f92c4c43 (127.11.182.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:41.145354 19141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:41.145603 19141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:41.145843 19141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:41.145889 19142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:41.146159 19142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:41.149818 19148 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:41.150259 19148 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:06:41.150525 19147 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c40684958ea42ad8974f33559cbcfe8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:41.150969 19147 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: This master's current role is: FOLLOWER
W20260929 02:06:41.179913 19140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:41.180195 19140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:41.180294 19140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:41.187565 19140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:41.187672 19140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:41.187790 19140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:41.197817 19140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab
--principal=oryx/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:34729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=42141
--tserver_master_addrs=127.11.182.254:44915,127.11.182.253:39003,127.11.182.252:41129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:41.199302 19140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:41.200562 19140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:41.214255 19156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:41.214330 19153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:41.214308 19154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:41.215284 19140 server_base.cc:1034] running on GCE node
I20260929 02:06:41.216041 19140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:41.217036 19140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:41.218278 19140 hybrid_clock.cc:648] HybridClock initialized: now 1790647601218219 us; error 47 us; skew 500 ppm
I20260929 02:06:41.218600 19140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:41 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647601, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:41.221798 19140 init.cc:377] Logged in from keytab as oryx/127.11.182.194@KRBTEST.COM (short username oryx)
I20260929 02:06:41.223389 19140 webserver.cc:460] Webserver started at http://127.11.182.194:42141/ using document root <none> and password file <none>
I20260929 02:06:41.224138 19140 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:41.224292 19140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:41.231264 19140 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:41.234448 19163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:41.235951 19140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:41.236135 19140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "2ed372e6334f4cf58de43c740cbddf2f"
format_stamp: "Formatted at 2026-09-29 02:06:38 on dist-test-slave-q06p"
server_key: "6184875d3e6fb50af853826ac5a4ca05"
server_key_iv: "5121848cf8a84943a7e8792f74bae157"
server_key_version: "encryptionkey@0"
I20260929 02:06:41.237115 19140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:41.252976 19140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:41.256904 19140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:41.257189 19140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:41.258131 19140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:41.259496 19140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:41.259629 19140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:41.259754 19140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:41.259845 19140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:41.294740 19140 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:34729
I20260929 02:06:41.294792 19275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:34729 every 8 connection(s)
I20260929 02:06:41.296375 19140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:41.304877 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 19140
I20260929 02:06:41.305160 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 18489
Sep 29 02:06:41 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647601, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.252@KRBTEST.COM
Sep 29 02:06:41 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647601, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
Sep 29 02:06:41 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647601, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.253@KRBTEST.COM
Sep 29 02:06:41 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.195@KRBTEST.COM
I20260929 02:06:41.325393 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.299198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:50871 (local address 127.11.182.254:44915)
0929 02:06:41.300377 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.300381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.301123 (+ 742us) server_negotiation.cc:407] Connection header received
0929 02:06:41.304572 (+ 3449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.304583 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.304705 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.304816 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.307991 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.308810 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.312993 (+ 4183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.313773 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.313943 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.317673 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:41.317682 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:41.317688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:41.317744 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:41.322443 (+ 4699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.323245 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.323264 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.323268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.323412 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.323869 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.323874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.323877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.324189 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:41.324356 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:41.324834 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 02:06:41.325031 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":921,"threads_started":1}
I20260929 02:06:41.326588 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.299939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.194:50871)
0929 02:06:41.300977 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:41.300993 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:41.304166 (+ 3173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:41.305411 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:41.305428 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:41.307178 (+ 1750us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:41.307789 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.307821 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.308983 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.308989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.311431 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.311445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.314000 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.314007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.314187 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.314805 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:41.314831 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:41.317480 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:41.322593 (+ 5113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.322601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.322607 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.323116 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.323559 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.323566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.323569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.323738 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.324431 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:41.324435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:41.324584 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 02:06:41.325613 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0929 02:06:41.325893 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":432,"spinlock_wait_cycles":11776,"thread_start_us":992,"threads_started":1}
I20260929 02:06:41.327602 19282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.299938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:41129 (local address 127.11.182.194:36485)
0929 02:06:41.300893 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:41.300937 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:41.304172 (+ 3235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:41.305882 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:41.305887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:41.306560 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:41.307787 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.307821 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.309672 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.309680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.311433 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.311446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.312493 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.312502 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.312795 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.313650 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:41.313685 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:41.316894 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:41.321494 (+ 4600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.321506 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.321527 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.322232 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.322681 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.322686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.322689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.322994 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.323640 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:41.323648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:41.324127 (+ 479us) client_negotiation.cc:769] Sending connection context
0929 02:06:41.324442 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 02:06:41.324991 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":71,"spinlock_wait_cycles":46336,"thread_start_us":461,"threads_started":1}
I20260929 02:06:41.328162 19277 heartbeater.cc:347] Connected to a master server at 127.11.182.254:44915
I20260929 02:06:41.328425 19277 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:41.329082 19277 heartbeater.cc:511] Master 127.11.182.254:44915 requested a full tablet report, sending...
I20260929 02:06:41.329847 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.299602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:36485 (local address 127.11.182.252:41129)
0929 02:06:41.300237 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.300244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.301070 (+ 826us) server_negotiation.cc:407] Connection header received
0929 02:06:41.305593 (+ 4523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.305601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.305691 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.306943 (+ 1252us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.308042 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.309459 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.311657 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.312236 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.312387 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.317177 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:41.317185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:41.317191 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:41.317231 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:41.321325 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.322409 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.322414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.322418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.322559 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.323147 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.323155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.323161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.323510 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:41.326612 (+ 3102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:41.326779 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 02:06:41.327044 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":338,"threads_started":1}
I20260929 02:06:41.330552 18160 ts_manager.cc:194] Re-registered known tserver with Master: 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729)
I20260929 02:06:41.331406 19276 heartbeater.cc:347] Connected to a master server at 127.11.182.252:41129
I20260929 02:06:41.331522 19276 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:41.331781 18160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:50871
I20260929 02:06:41.332485 19276 heartbeater.cc:511] Master 127.11.182.252:41129 requested a full tablet report, sending...
I20260929 02:06:41.333150 19285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.301888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:48743 (local address 127.11.182.253:39003)
0929 02:06:41.306827 (+ 4939us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.306832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.306896 (+ 64us) server_negotiation.cc:407] Connection header received
0929 02:06:41.306963 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.306969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.307064 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.307201 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.311349 (+ 4148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.312821 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.318602 (+ 5781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.319318 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.319469 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.324439 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:41.324446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:41.324451 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:41.324507 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:41.328841 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.330588 (+ 1747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.330594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.330597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.330745 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.331231 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.331237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.331240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.331593 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:41.331768 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:41.332575 (+ 807us) server_negotiation.cc:299] Negotiation successful
0929 02:06:41.332813 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4837,"thread_start_us":4707,"threads_started":1}
I20260929 02:06:41.333885 18949 ts_manager.cc:194] Registered new tserver with Master: 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729)
I20260929 02:06:41.334102 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.301739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.11.182.194:48743)
0929 02:06:41.302283 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:41.302300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:41.304171 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:41.307599 (+ 3428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:41.307605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:41.309465 (+ 1860us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:41.311138 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.311154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.314258 (+ 3104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.314264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.317499 (+ 3235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.317520 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.319831 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.319840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.320083 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.320706 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:41.320737 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:41.324186 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:41.329929 (+ 5743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.329936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.329942 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.330430 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.330929 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.330934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.330938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.331092 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.331868 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:41.331872 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:41.332265 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 02:06:41.333333 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0929 02:06:41.333709 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":1252,"thread_start_us":292,"threads_started":1}
I20260929 02:06:41.334859 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.306436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.11.182.195:41601)
0929 02:06:41.310037 (+ 3601us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.310047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.310575 (+ 528us) server_negotiation.cc:407] Connection header received
0929 02:06:41.310742 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.310751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.311023 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.311174 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.313655 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.315531 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.317574 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.318514 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.318655 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.325735 (+ 7080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:41.325755 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:41.325774 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:41.325821 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:41.329717 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.330942 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.330956 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.330959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.331137 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.331673 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.331678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.331685 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.332562 (+ 877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:41.332852 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:41.334135 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0929 02:06:41.334439 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3418,"thread_start_us":3268,"threads_started":1}
I20260929 02:06:41.335413 19278 heartbeater.cc:347] Connected to a master server at 127.11.182.253:39003
I20260929 02:06:41.335554 19278 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:41.335954 19278 heartbeater.cc:511] Master 127.11.182.253:39003 requested a full tablet report, sending...
I20260929 02:06:41.336230 18612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:41.336993 18758 ts_manager.cc:194] Registered new tserver with Master: 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729)
I20260929 02:06:41.445432 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 18489
I20260929 02:06:41.462625 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:41601
--local_ip_for_outbound_sockets=127.11.182.195
--tserver_master_addrs=127.11.182.254:44915
--webserver_port=38581
--webserver_interface=127.11.182.195
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab
--principal=oryx/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.182.254:44915,127.11.182.253:39003
--tserver_master_addrs=127.11.182.254:44915,127.11.182.253:39003,127.11.182.252:41129 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:06:41.747493 19129 heartbeater.cc:503] Master 127.11.182.254:44915 was elected leader, sending a full tablet report...
W20260929 02:06:41.806164 19289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:41.806473 19289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:41.806633 19289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:41.814066 19289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:41.814215 19289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:41.814383 19289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:41.824105 19289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab
--principal=oryx/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:41601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=38581
--tserver_master_addrs=127.11.182.254:44915,127.11.182.253:39003,127.11.182.252:41129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:41.825634 19289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:41.827217 19289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:41.840618 19294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:41.840659 19297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:41.840683 19295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:41.841409 19289 server_base.cc:1034] running on GCE node
I20260929 02:06:41.841977 19289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:41.842974 19289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:41.844185 19289 hybrid_clock.cc:648] HybridClock initialized: now 1790647601844142 us; error 45 us; skew 500 ppm
I20260929 02:06:41.844563 19289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:41 dist-test-slave-q06p krb5kdc[18116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647601, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:41.847818 19289 init.cc:377] Logged in from keytab as oryx/127.11.182.195@KRBTEST.COM (short username oryx)
I20260929 02:06:41.849310 19289 webserver.cc:460] Webserver started at http://127.11.182.195:38581/ using document root <none> and password file <none>
I20260929 02:06:41.849983 19289 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:41.850128 19289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:41.856058 19289 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:06:41.858747 19304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:41.860172 19289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:06:41.860351 19289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "b81b0df2be8b4d5cb7b042ee400726d2"
format_stamp: "Formatted at 2026-09-29 02:06:38 on dist-test-slave-q06p"
server_key: "9aa6cd5a8de1c97bfc8ee0f2fa54444c"
server_key_iv: "1fb8eea92fe780082c47df323996a46d"
server_key_version: "encryptionkey@0"
I20260929 02:06:41.861387 19289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:41.886380 19289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:41.891315 19289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:41.891624 19289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:41.892606 19289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:41.894083 19289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:41.894204 19289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:41.894332 19289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:41.894423 19289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:41.929529 19289 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:41601
I20260929 02:06:41.929618 19416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:41601 every 8 connection(s)
I20260929 02:06:41.931017 19289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:41.935427 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 19289
Sep 29 02:06:41 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647601, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.252@KRBTEST.COM
Sep 29 02:06:41 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647601, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.253@KRBTEST.COM
Sep 29 02:06:41 dist-test-slave-q06p krb5kdc[18116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647601, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:41.977613 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.945818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:37731 (local address 127.11.182.253:39003)
0929 02:06:41.951672 (+ 5854us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.951679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.952609 (+ 930us) server_negotiation.cc:407] Connection header received
0929 02:06:41.952784 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.952790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.952894 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.953001 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.955873 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.957182 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.959944 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.960661 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.960779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.968433 (+ 7654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:41.968439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:41.968444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:41.968490 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:41.972279 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.973462 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.973468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.973475 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.973641 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.974244 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.974251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.974256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.974630 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:41.974778 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:41.976897 (+ 2119us) server_negotiation.cc:299] Negotiation successful
0929 02:06:41.977161 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5727,"thread_start_us":5602,"threads_started":1}
I20260929 02:06:41.981741 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.945589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.11.182.195:37731)
0929 02:06:41.952494 (+ 6905us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:41.952509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:41.952664 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:41.953196 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:41.953200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:41.954152 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:41.955634 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.955682 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.957385 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.957390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.959757 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.959772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.961069 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.961076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.961392 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.962239 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:41.962274 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:41.968187 (+ 5913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:41.972468 (+ 4281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.972480 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.972503 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.973239 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.973783 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.973790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.973794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.974067 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.974914 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:41.974925 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:41.975458 (+ 533us) client_negotiation.cc:769] Sending connection context
0929 02:06:41.975786 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 02:06:41.976369 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6801,"mutex_wait_us":185,"spinlock_wait_cycles":33920,"thread_start_us":6656,"threads_started":1}
I20260929 02:06:41.983630 19419 heartbeater.cc:347] Connected to a master server at 127.11.182.253:39003
I20260929 02:06:41.983733 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.937520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.195:40393)
0929 02:06:41.948548 (+ 11028us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:41.948574 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:41.951433 (+ 2859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:41.952407 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:41.952422 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:41.954201 (+ 1779us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:41.955629 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.955680 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.957374 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.957382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.964087 (+ 6705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.964107 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.965171 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.965177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.965568 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.966359 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:41.966377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:41.971985 (+ 5608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:41.980486 (+ 8501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.980492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.980497 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.981011 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.981797 (+ 786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.981802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.981806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.981968 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.982651 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:41.982656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:41.982838 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 02:06:41.983002 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 02:06:41.983303 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10866,"mutex_wait_us":4015,"spinlock_wait_cycles":28928,"thread_start_us":6566,"threads_started":1}
I20260929 02:06:41.983984 19419 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:41.984719 19419 heartbeater.cc:511] Master 127.11.182.253:39003 requested a full tablet report, sending...
I20260929 02:06:41.984932 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.934155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:40393 (local address 127.11.182.254:44915)
0929 02:06:41.948023 (+ 13868us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.948032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.948775 (+ 743us) server_negotiation.cc:407] Connection header received
0929 02:06:41.951835 (+ 3060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.951842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.951929 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.952037 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.955917 (+ 3880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.957163 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.964259 (+ 7096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.964947 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.965125 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.972210 (+ 7085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:41.972220 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:41.972228 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:41.972280 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:41.978202 (+ 5922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.981174 (+ 2972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.981182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.981188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.981371 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.982163 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.982169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.982172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.982519 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:41.982676 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:41.984279 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0929 02:06:41.984534 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13700,"thread_start_us":13560,"threads_started":1}
I20260929 02:06:41.985224 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.934170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:38793 (local address 127.11.182.252:41129)
0929 02:06:41.947485 (+ 13315us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.947494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.947526 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:06:41.952918 (+ 5392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.952928 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.953031 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.953144 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.957554 (+ 4410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.958696 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.960579 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.961213 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.961308 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.968160 (+ 6852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:41.968167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:41.968172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:41.968220 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:41.977032 (+ 8812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.979385 (+ 2353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.979395 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.979401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.979591 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.980560 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.980567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.980573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.980950 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:41.981091 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:41.983097 (+ 2006us) server_negotiation.cc:299] Negotiation successful
0929 02:06:41.983400 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13146,"thread_start_us":13344,"threads_started":1}
I20260929 02:06:41.986954 19418 heartbeater.cc:347] Connected to a master server at 127.11.182.254:44915
I20260929 02:06:41.987051 18758 ts_manager.cc:194] Registered new tserver with Master: b81b0df2be8b4d5cb7b042ee400726d2 (127.11.182.195:41601)
I20260929 02:06:41.987111 19418 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:41.987617 19418 heartbeater.cc:511] Master 127.11.182.254:44915 requested a full tablet report, sending...
I20260929 02:06:41.988780 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.934945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:41129 (local address 127.11.182.195:38793)
0929 02:06:41.944750 (+ 9805us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:41.944829 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:41.952773 (+ 7944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:41.953231 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:41.953238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:41.954261 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:41.957350 (+ 3089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.957367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.958918 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.958923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.960415 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:41.960430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.961582 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.961589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:41.962147 (+ 558us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.962807 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:41.962830 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:41.967812 (+ 4982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:41.978407 (+ 10595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.978416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.978423 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.979183 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.979782 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:41.979790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:41.979797 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:41.980423 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:41.981553 (+ 1130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:41.981561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:41.981784 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 02:06:41.985608 (+ 3824us) client_negotiation.cc:240] Negotiation successful
0929 02:06:41.986058 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9304,"mutex_wait_us":1994,"spinlock_wait_cycles":11520,"thread_start_us":15822,"threads_started":1}
I20260929 02:06:41.989153 18160 ts_manager.cc:194] Re-registered known tserver with Master: b81b0df2be8b4d5cb7b042ee400726d2 (127.11.182.195:41601)
I20260929 02:06:41.990399 18160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:40393
I20260929 02:06:41.991622 19417 heartbeater.cc:347] Connected to a master server at 127.11.182.252:41129
I20260929 02:06:41.991760 19417 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:41.992446 19417 heartbeater.cc:511] Master 127.11.182.252:41129 requested a full tablet report, sending...
I20260929 02:06:41.993592 18949 ts_manager.cc:194] Registered new tserver with Master: b81b0df2be8b4d5cb7b042ee400726d2 (127.11.182.195:41601)
I20260929 02:06:41.998884 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.958228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.11.182.254:44915)
0929 02:06:41.971442 (+ 13214us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.971448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.971470 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:06:41.971530 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.971538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.971641 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.974727 (+ 3086us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.975020 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.976305 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.978452 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.979106 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.979218 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.989408 (+ 10190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:41.989414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:41.989417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:41.989449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:41.994117 (+ 4668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.995161 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.995168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.995173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.995352 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:41.995839 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:41.995844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:41.995847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:41.996257 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:41.996933 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:41.998127 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0929 02:06:41.998401 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12287,"thread_start_us":12199,"threads_started":1}
I20260929 02:06:42.004366 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.975620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.11.182.253:39003)
0929 02:06:41.978228 (+ 2608us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.978238 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.988658 (+ 10420us) server_negotiation.cc:407] Connection header received
0929 02:06:41.989081 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.989086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.989194 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.989267 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.991221 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.992286 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.994581 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.995185 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.995304 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:41.998753 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:41.998759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:41.998765 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:41.998806 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:42.001712 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.002379 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.002385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.002389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.002572 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.003001 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.003007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.003010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.003269 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:42.003562 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.003845 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.004069 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2440,"thread_start_us":6688,"threads_started":1}
I20260929 02:06:42.006835 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:41.958561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.11.182.252:41129)
0929 02:06:41.971008 (+ 12447us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:41.971016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:41.974551 (+ 3535us) server_negotiation.cc:407] Connection header received
0929 02:06:41.974628 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:41.974633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:41.974720 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:41.974828 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:41.987039 (+ 12211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.991946 (+ 4907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.994535 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:41.995267 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:41.996155 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.000175 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:42.000181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:42.000187 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:42.000226 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:42.003662 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.004507 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.004512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.004516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.004733 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.005159 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.005166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.005171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.005521 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:42.005927 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.006290 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.006485 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11092,"thread_start_us":10900,"threads_started":1}
I20260929 02:06:42.003600 18160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:42.012497 18160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:42.048178 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.036836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.11.182.195:41601)
0929 02:06:42.037784 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.037797 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.038235 (+ 438us) server_negotiation.cc:407] Connection header received
0929 02:06:42.039197 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.039210 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.039606 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.040885 (+ 1279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:42.041280 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.043433 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.045659 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.047122 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.047309 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.047494 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.047719 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.047875 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":467,"threads_started":1}
I20260929 02:06:42.048247 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.036436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:41601 (local address 127.0.0.1:43854)
0929 02:06:42.038039 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:42.038058 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:42.038976 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:42.039982 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:42.039990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:42.040030 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:42.040518 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.040538 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.043584 (+ 3046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.043590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.045506 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.045517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.047322 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.047328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.047484 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.047529 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 02:06:42.047653 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 02:06:42.047794 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453,"mutex_wait_us":1,"thread_start_us":1328,"threads_started":1}
I20260929 02:06:42.052951 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.034314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:34729 (local address 127.0.0.1:45814)
0929 02:06:42.035248 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:42.035269 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:42.035369 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:42.044267 (+ 8898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:42.044275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:42.044320 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:42.044863 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.044879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.048181 (+ 3302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.048190 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.050236 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.050251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.052125 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.052131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.052318 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.052355 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:06:42.052457 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 02:06:42.052612 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":717,"threads_started":1}
I20260929 02:06:42.052961 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.034558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.11.182.194:34729)
0929 02:06:42.043395 (+ 8837us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.043409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.043449 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:06:42.043587 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.043598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.043977 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.045120 (+ 1143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:42.045361 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.047320 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.050423 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.051905 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.052081 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.052270 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.052513 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.052683 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8616,"thread_start_us":8474,"threads_started":1}
I20260929 02:06:42.052649 19352 tablet_service.cc:1467] Processing CreateTablet for tablet 4359dc10c3c34d1195ddd3134d53e199 (DEFAULT_TABLE table=test-table [id=7aab1d0a58d24f3ca21fa265e7f0f949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:42.054199 19352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4359dc10c3c34d1195ddd3134d53e199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:42.056890 19211 tablet_service.cc:1467] Processing CreateTablet for tablet 4359dc10c3c34d1195ddd3134d53e199 (DEFAULT_TABLE table=test-table [id=7aab1d0a58d24f3ca21fa265e7f0f949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:42.058218 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4359dc10c3c34d1195ddd3134d53e199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:42.071403 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.038095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:35255 (local address 127.0.0.1:43842)
0929 02:06:42.047916 (+ 9821us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:42.047935 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:42.048027 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:42.064095 (+ 16068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:42.064100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:42.064128 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:42.064624 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.064637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.066546 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.066551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.068749 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.068767 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.070518 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.070525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.070728 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.070767 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:06:42.070909 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 02:06:42.071057 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9706,"thread_start_us":9506,"threads_started":1}
I20260929 02:06:42.071425 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.038373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.11.182.193:35255)
0929 02:06:42.039026 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.039038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.063469 (+ 24431us) server_negotiation.cc:407] Connection header received
0929 02:06:42.063618 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.063626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.063900 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.064110 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:42.064815 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.066321 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.068904 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.070285 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.070467 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.070650 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.070982 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.071142 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":301,"threads_started":1}
I20260929 02:06:42.075414 19451 tablet_bootstrap.cc:492] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: Bootstrap starting.
I20260929 02:06:42.076507 19450 tablet_bootstrap.cc:492] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: Bootstrap starting.
I20260929 02:06:42.077278 19063 tablet_service.cc:1467] Processing CreateTablet for tablet 4359dc10c3c34d1195ddd3134d53e199 (DEFAULT_TABLE table=test-table [id=7aab1d0a58d24f3ca21fa265e7f0f949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:42.079113 19451 tablet_bootstrap.cc:654] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:42.079437 19063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4359dc10c3c34d1195ddd3134d53e199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:42.080232 19451 log.cc:824] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:42.080684 19450 tablet_bootstrap.cc:654] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:42.081833 19450 log.cc:824] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:42.083592 19451 tablet_bootstrap.cc:492] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: No bootstrap required, opened a new log
I20260929 02:06:42.083871 19451 ts_tablet_manager.cc:1397] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 02:06:42.090062 19450 tablet_bootstrap.cc:492] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: No bootstrap required, opened a new log
I20260929 02:06:42.090389 19450 ts_tablet_manager.cc:1397] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260929 02:06:42.091869 19451 raft_consensus.cc:348] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } }
I20260929 02:06:42.092221 19451 raft_consensus.cc:374] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:42.092319 19451 raft_consensus.cc:729] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed372e6334f4cf58de43c740cbddf2f, State: Initialized, Role: FOLLOWER
I20260929 02:06:42.092938 19451 consensus_queue.cc:260] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } }
I20260929 02:06:42.094691 19451 ts_tablet_manager.cc:1428] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260929 02:06:42.094969 19277 heartbeater.cc:503] Master 127.11.182.254:44915 was elected leader, sending a full tablet report...
I20260929 02:06:42.103607 19450 raft_consensus.cc:348] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } }
I20260929 02:06:42.104255 19450 raft_consensus.cc:374] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:42.104378 19450 raft_consensus.cc:729] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81b0df2be8b4d5cb7b042ee400726d2, State: Initialized, Role: FOLLOWER
I20260929 02:06:42.105278 19450 consensus_queue.cc:260] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } }
I20260929 02:06:42.106573 19455 tablet_bootstrap.cc:492] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: Bootstrap starting.
I20260929 02:06:42.108021 19418 heartbeater.cc:503] Master 127.11.182.254:44915 was elected leader, sending a full tablet report...
I20260929 02:06:42.110024 19455 tablet_bootstrap.cc:654] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:42.111114 19455 log.cc:824] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:42.111990 19450 ts_tablet_manager.cc:1428] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: Time spent starting tablet: real 0.021s user 0.008s sys 0.008s
I20260929 02:06:42.114953 19455 tablet_bootstrap.cc:492] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: No bootstrap required, opened a new log
I20260929 02:06:42.115231 19455 ts_tablet_manager.cc:1397] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260929 02:06:42.122197 19455 raft_consensus.cc:348] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } }
I20260929 02:06:42.122542 19455 raft_consensus.cc:374] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:42.122658 19455 raft_consensus.cc:729] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c9eede04a541b1b1f2dd6b37a882a1, State: Initialized, Role: FOLLOWER
I20260929 02:06:42.123246 19455 consensus_queue.cc:260] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } }
I20260929 02:06:42.124837 19455 ts_tablet_manager.cc:1428] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260929 02:06:42.184243 19420 tablet.cc:2369] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:06:42.201999 19131 tablet.cc:2369] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:06:42.254654 19456 raft_consensus.cc:482] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:42.254997 19456 raft_consensus.cc:504] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } }
I20260929 02:06:42.257009 19456 leader_election.cc:288] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729), 27c9eede04a541b1b1f2dd6b37a882a1 (127.11.182.193:35255)
I20260929 02:06:42.263010 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.257438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:56523 (local address 127.11.182.194:34729)
0929 02:06:42.257663 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.257670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.257693 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:06:42.257888 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.257895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.257975 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.258065 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:42.258473 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.259459 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.261174 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.262001 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.262145 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.262223 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.262586 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.262720 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:06:42.263018 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.257889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:57989 (local address 127.11.182.193:35255)
0929 02:06:42.258106 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.258111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.258127 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:06:42.258190 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.258194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.258276 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.258377 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:42.258733 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.259704 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.261296 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.262023 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.262186 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.262270 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.262654 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.262769 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:06:42.263025 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.257301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:34729 (local address 127.11.182.195:56523)
0929 02:06:42.257511 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:42.257533 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:42.257683 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:42.258104 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:42.258110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:42.258144 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:42.258338 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.258354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.259594 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.259599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.261016 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.261028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.262157 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.262163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.262416 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.262454 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:06:42.262527 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 02:06:42.262675 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":24}
I20260929 02:06:42.263031 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.257737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:35255 (local address 127.11.182.195:57989)
0929 02:06:42.257872 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:42.257889 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:42.257986 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:42.258398 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:42.258402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:42.258446 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:42.258602 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.258612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.259846 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.259850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.261123 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:42.261136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.262209 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.262215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:42.262468 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.262495 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 02:06:42.262610 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 02:06:42.262724 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":37}
I20260929 02:06:42.264285 19083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4359dc10c3c34d1195ddd3134d53e199" candidate_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" is_pre_election: true
I20260929 02:06:42.264279 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4359dc10c3c34d1195ddd3134d53e199" candidate_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed372e6334f4cf58de43c740cbddf2f" is_pre_election: true
I20260929 02:06:42.264726 19231 raft_consensus.cc:2454] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81b0df2be8b4d5cb7b042ee400726d2 in term 0.
I20260929 02:06:42.264747 19083 raft_consensus.cc:2454] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81b0df2be8b4d5cb7b042ee400726d2 in term 0.
I20260929 02:06:42.265396 19308 leader_election.cc:302] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ed372e6334f4cf58de43c740cbddf2f, b81b0df2be8b4d5cb7b042ee400726d2; no voters:
I20260929 02:06:42.265769 19456 raft_consensus.cc:2781] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:42.265890 19456 raft_consensus.cc:482] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:42.266038 19456 raft_consensus.cc:3037] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:42.268653 19456 raft_consensus.cc:504] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } }
I20260929 02:06:42.269176 19456 leader_election.cc:288] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [CANDIDATE]: Term 1 election: Requested vote from peers 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729), 27c9eede04a541b1b1f2dd6b37a882a1 (127.11.182.193:35255)
I20260929 02:06:42.269670 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4359dc10c3c34d1195ddd3134d53e199" candidate_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed372e6334f4cf58de43c740cbddf2f"
I20260929 02:06:42.269702 19083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4359dc10c3c34d1195ddd3134d53e199" candidate_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c9eede04a541b1b1f2dd6b37a882a1"
I20260929 02:06:42.269898 19083 raft_consensus.cc:3037] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:42.269893 19231 raft_consensus.cc:3037] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:42.273514 19083 raft_consensus.cc:2454] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81b0df2be8b4d5cb7b042ee400726d2 in term 1.
I20260929 02:06:42.273514 19231 raft_consensus.cc:2454] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81b0df2be8b4d5cb7b042ee400726d2 in term 1.
I20260929 02:06:42.273949 19308 leader_election.cc:302] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ed372e6334f4cf58de43c740cbddf2f, b81b0df2be8b4d5cb7b042ee400726d2; no voters:
I20260929 02:06:42.274236 19456 raft_consensus.cc:2781] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:42.274638 19456 raft_consensus.cc:686] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 LEADER]: Becoming Leader. State: Replica: b81b0df2be8b4d5cb7b042ee400726d2, State: Running, Role: LEADER
I20260929 02:06:42.275292 19456 consensus_queue.cc:237] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } }
I20260929 02:06:42.280375 18161 catalog_manager.cc:5495] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b81b0df2be8b4d5cb7b042ee400726d2 (127.11.182.195). New cstate: current_term: 1 leader_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } health_report { overall_health: UNKNOWN } } }
W20260929 02:06:42.300781 19279 tablet.cc:2369] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:06:42.340045 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.334643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.11.182.195:41601)
0929 02:06:42.334925 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.334930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.334953 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:06:42.335149 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.335153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.335229 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.335366 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:42.335747 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.336581 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.337905 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.338295 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.338390 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.338930 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:42.339457 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:42.339564 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.339664 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.339727 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:06:42.348145 19083 raft_consensus.cc:1261] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 1 FOLLOWER]: Refusing update from remote peer b81b0df2be8b4d5cb7b042ee400726d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:42.348194 19231 raft_consensus.cc:1261] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [term 1 FOLLOWER]: Refusing update from remote peer b81b0df2be8b4d5cb7b042ee400726d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:42.349078 19459 consensus_queue.cc:1035] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:42.349439 19456 consensus_queue.cc:1035] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:42.364538 19462 mvcc.cc:205] Tried to move back new op lower bound from 7334492579208507392 to 7334492578923339776. Current Snapshot: MvccSnapshot[applied={T|T < 7334492579208507392}]
I20260929 02:06:42.373267 19464 mvcc.cc:205] Tried to move back new op lower bound from 7334492579208507392 to 7334492578923339776. Current Snapshot: MvccSnapshot[applied={T|T < 7334492579208507392 or (T in {7334492579208507392})}]
I20260929 02:06:42.379897 18161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:42.380169 18161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:42.385914 18161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c40684958ea42ad8974f33559cbcfe8: Sending DeleteTablet for 3 replicas of tablet 4359dc10c3c34d1195ddd3134d53e199
I20260929 02:06:42.388347 19211 tablet_service.cc:1514] Processing DeleteTablet for tablet 4359dc10c3c34d1195ddd3134d53e199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:42 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:45814
I20260929 02:06:42.388427 19063 tablet_service.cc:1514] Processing DeleteTablet for tablet 4359dc10c3c34d1195ddd3134d53e199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:42 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:43842
I20260929 02:06:42.388548 19352 tablet_service.cc:1514] Processing DeleteTablet for tablet 4359dc10c3c34d1195ddd3134d53e199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:42 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:43854
I20260929 02:06:42.392714 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 18129
I20260929 02:06:42.393877 18191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:42.401635 19471 tablet_replica.cc:330] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: stopping tablet replica
I20260929 02:06:42.402329 19473 tablet_replica.cc:330] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: stopping tablet replica
I20260929 02:06:42.403105 19473 raft_consensus.cc:2229] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:42.403764 19471 raft_consensus.cc:2229] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:42.404481 19471 raft_consensus.cc:2258] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:42.405287 19473 raft_consensus.cc:2258] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:42.408366 19471 ts_tablet_manager.cc:1905] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:42.408459 19472 tablet_replica.cc:330] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: stopping tablet replica
I20260929 02:06:42.408901 19472 raft_consensus.cc:2229] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:42.409520 19472 raft_consensus.cc:2258] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:42.410796 19473 ts_tablet_manager.cc:1905] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:42.412158 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.390555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.11.182.253:39003)
0929 02:06:42.390777 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.390786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.391557 (+ 771us) server_negotiation.cc:407] Connection header received
0929 02:06:42.391861 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.391870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.391968 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.392065 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:42.395711 (+ 3646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.397004 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.398974 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.399649 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.399746 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.404210 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:42.404221 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:42.404228 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:42.404282 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:42.408701 (+ 4419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.409587 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.409593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.409596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.409749 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.410219 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.410226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.410231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.410616 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:42.410880 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.411459 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.411691 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 02:06:42.414613 19472 ts_tablet_manager.cc:1905] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:42.427670 19472 ts_tablet_manager.cc:1918] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:42.430768 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.392060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.11.182.252:41129)
0929 02:06:42.392240 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.392246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.413644 (+ 21398us) server_negotiation.cc:407] Connection header received
0929 02:06:42.413887 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.413893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.413981 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.414065 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:42.415930 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.417015 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.418786 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.419553 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.419653 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.423250 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:42.423258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:42.423265 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:42.423308 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:42.426733 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.427635 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.427642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.427648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.427785 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.428334 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.428342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.428348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.428715 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:42.429381 (+ 666us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.430172 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.430435 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:06:42.431329 19473 ts_tablet_manager.cc:1918] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:42.431551 19473 log.cc:1196] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/4359dc10c3c34d1195ddd3134d53e199
I20260929 02:06:42.427857 19472 log.cc:1196] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/4359dc10c3c34d1195ddd3134d53e199
I20260929 02:06:42.430748 19471 ts_tablet_manager.cc:1918] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:42.432145 19472 ts_tablet_manager.cc:1939] T 4359dc10c3c34d1195ddd3134d53e199 P 27c9eede04a541b1b1f2dd6b37a882a1: Deleting consensus metadata
I20260929 02:06:42.432344 19471 log.cc:1196] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/4359dc10c3c34d1195ddd3134d53e199
I20260929 02:06:42.432894 19471 ts_tablet_manager.cc:1939] T 4359dc10c3c34d1195ddd3134d53e199 P 2ed372e6334f4cf58de43c740cbddf2f: Deleting consensus metadata
I20260929 02:06:42.432179 19473 ts_tablet_manager.cc:1939] T 4359dc10c3c34d1195ddd3134d53e199 P b81b0df2be8b4d5cb7b042ee400726d2: Deleting consensus metadata
I20260929 02:06:42.547259 18148 catalog_manager.cc:4841] TS 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729): tablet 4359dc10c3c34d1195ddd3134d53e199 (table test-table [id=7aab1d0a58d24f3ca21fa265e7f0f949]) successfully deleted
I20260929 02:06:42.547766 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 18129
I20260929 02:06:42.547727 18148 catalog_manager.cc:4841] TS b81b0df2be8b4d5cb7b042ee400726d2 (127.11.182.195:41601): tablet 4359dc10c3c34d1195ddd3134d53e199 (table test-table [id=7aab1d0a58d24f3ca21fa265e7f0f949]) successfully deleted
W20260929 02:06:42.577382 19016 connection.cc:535] server connection from 127.0.0.1:43842 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:06:42.599188 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.582177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.11.182.252:41129)
0929 02:06:42.582362 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.582369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.583897 (+ 1528us) server_negotiation.cc:407] Connection header received
0929 02:06:42.584185 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.584192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.584291 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.584390 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:42.586791 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.587929 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.589613 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.590189 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.590324 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.593826 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:42.593833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:42.593839 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:42.593883 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:42.596470 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.597157 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.597163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.597167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.597298 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.597697 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.597703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.597706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.597994 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:42.598256 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.598588 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.598780 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:06:42.600054 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:42.584084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.11.182.253:39003)
0929 02:06:42.584253 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:42.584258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:42.584382 (+ 124us) server_negotiation.cc:407] Connection header received
0929 02:06:42.584521 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:42.584527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:42.584600 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:42.584701 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:42.586866 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.587580 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.589338 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:42.590031 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:42.590160 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:42.593127 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:42.593133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:42.593138 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:42.593181 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:42.596502 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.597346 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.597351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.597355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.597627 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:42.598031 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:42.598036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:42.598040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:42.598383 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:42.598586 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:42.599201 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 02:06:42.599426 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:06:43.373143 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.372222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.195:41591)
0929 02:06:43.372738 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.372899 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":282,"threads_started":1}
W20260929 02:06:43.373733 19418 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:44915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
I20260929 02:06:43.376189 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.375187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.194:49501)
0929 02:06:43.375879 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.376008 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":428,"threads_started":1}
W20260929 02:06:43.376907 19277 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:44915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
I20260929 02:06:43.378183 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.376356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.193:38135)
0929 02:06:43.377375 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.378005 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":822,"threads_started":1}
W20260929 02:06:43.378501 19129 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:44915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
I20260929 02:06:43.887655 19485 raft_consensus.cc:482] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c40684958ea42ad8974f33559cbcfe8)
I20260929 02:06:43.887962 19485 raft_consensus.cc:504] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } }
I20260929 02:06:43.888332 19486 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c40684958ea42ad8974f33559cbcfe8)
I20260929 02:06:43.888571 19486 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } }
I20260929 02:06:43.889467 19485 leader_election.cc:288] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c40684958ea42ad8974f33559cbcfe8 (127.11.182.254:44915), fbda9dd051364eadbc87bc57ccd4289a (127.11.182.253:39003)
I20260929 02:06:43.889787 19486 leader_election.cc:288] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c40684958ea42ad8974f33559cbcfe8 (127.11.182.254:44915), bafae58e078f44b9bc3fc977f92c4c43 (127.11.182.252:41129)
I20260929 02:06:43.890872 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.889663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56872)
0929 02:06:43.890439 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.890593 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":439,"threads_started":1}
I20260929 02:06:43.892473 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.889893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56886)
0929 02:06:43.891901 (+ 2008us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.892140 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1854,"threads_started":1}
W20260929 02:06:43.893777 18743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
W20260929 02:06:43.893910 18933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
I20260929 02:06:43.901948 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.890804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.11.182.253:39003)
0929 02:06:43.892298 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:43.892307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:43.892333 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:06:43.892420 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:43.892428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:43.892529 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:43.893468 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:43.893663 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.895408 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.898666 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.900570 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.900711 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.900809 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:43.901378 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 02:06:43.901513 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":1323,"thread_start_us":1073,"threads_started":1}
I20260929 02:06:43.901999 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.901454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56890)
0929 02:06:43.901662 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.901817 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 02:06:43.902235 18743 leader_election.cc:334] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c40684958ea42ad8974f33559cbcfe8 (127.11.182.254:44915): Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
I20260929 02:06:43.902386 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.890505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.0.0.1:40460)
0929 02:06:43.891091 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.891112 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:43.891258 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:43.892849 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:43.892920 (+ 71us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:43.892957 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:43.893307 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.893321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.895619 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.895628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.898391 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.898410 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.900913 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.900919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.901126 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.901165 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:06:43.901276 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:06:43.901624 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":1,"thread_start_us":2609,"threads_started":1}
I20260929 02:06:43.903141 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bafae58e078f44b9bc3fc977f92c4c43" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbda9dd051364eadbc87bc57ccd4289a" is_pre_election: true
I20260929 02:06:43.903493 18768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bafae58e078f44b9bc3fc977f92c4c43 in term 1.
I20260929 02:06:43.903682 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.903317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56900)
0929 02:06:43.903444 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.903522 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 02:06:43.903947 18933 leader_election.cc:334] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c40684958ea42ad8974f33559cbcfe8 (127.11.182.254:44915): Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
I20260929 02:06:43.904078 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.889928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:41129 (local address 127.0.0.1:57248)
0929 02:06:43.892698 (+ 2770us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.892718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:43.892865 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:43.894237 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:43.894243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:43.894275 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:43.894775 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.894790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.897104 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.897112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.900537 (+ 3425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.900553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.903310 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.903316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.903494 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.903527 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:06:43.903613 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:06:43.903766 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2653,"mutex_wait_us":467,"thread_start_us":1661,"threads_started":1}
I20260929 02:06:43.904213 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.893372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.11.182.252:41129)
0929 02:06:43.893919 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:43.893927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:43.893945 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:06:43.894008 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:43.894016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:43.894082 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:43.894970 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:43.895270 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.896916 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.900731 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.903110 (+ 2379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.903274 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.903345 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:43.903682 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 02:06:43.903839 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":314,"threads_started":1}
I20260929 02:06:43.904352 18933 leader_election.cc:302] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bafae58e078f44b9bc3fc977f92c4c43, fbda9dd051364eadbc87bc57ccd4289a; no voters: 6c40684958ea42ad8974f33559cbcfe8
I20260929 02:06:43.904678 19485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 02:06:43.904798 19485 raft_consensus.cc:482] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c40684958ea42ad8974f33559cbcfe8)
I20260929 02:06:43.904907 19485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:06:43.905243 18959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbda9dd051364eadbc87bc57ccd4289a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bafae58e078f44b9bc3fc977f92c4c43" is_pre_election: true
I20260929 02:06:43.907054 19485 raft_consensus.cc:504] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } }
I20260929 02:06:43.907457 18959 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fbda9dd051364eadbc87bc57ccd4289a in current term 2: Already voted for candidate bafae58e078f44b9bc3fc977f92c4c43 in this term.
I20260929 02:06:43.907656 19485 leader_election.cc:288] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [CANDIDATE]: Term 2 election: Requested vote from peers 6c40684958ea42ad8974f33559cbcfe8 (127.11.182.254:44915), fbda9dd051364eadbc87bc57ccd4289a (127.11.182.253:39003)
I20260929 02:06:43.908149 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.907783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56908)
0929 02:06:43.907902 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.907964 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 02:06:43.908329 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bafae58e078f44b9bc3fc977f92c4c43" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbda9dd051364eadbc87bc57ccd4289a"
I20260929 02:06:43.908458 18742 leader_election.cc:302] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbda9dd051364eadbc87bc57ccd4289a; no voters: 6c40684958ea42ad8974f33559cbcfe8, bafae58e078f44b9bc3fc977f92c4c43
I20260929 02:06:43.908552 18768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:06:43.909075 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.908712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56914)
0929 02:06:43.908840 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.908937 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 02:06:43.909255 18933 leader_election.cc:334] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c40684958ea42ad8974f33559cbcfe8 (127.11.182.254:44915): Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
I20260929 02:06:43.910601 18768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bafae58e078f44b9bc3fc977f92c4c43 in term 2.
I20260929 02:06:43.910758 19486 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:06:43.911072 18933 leader_election.cc:302] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bafae58e078f44b9bc3fc977f92c4c43, fbda9dd051364eadbc87bc57ccd4289a; no voters: 6c40684958ea42ad8974f33559cbcfe8
I20260929 02:06:43.911346 19485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:06:43.911743 19485 raft_consensus.cc:686] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [term 2 LEADER]: Becoming Leader. State: Replica: bafae58e078f44b9bc3fc977f92c4c43, State: Running, Role: LEADER
I20260929 02:06:43.912099 19485 consensus_queue.cc:237] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } }
I20260929 02:06:43.913781 19495 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bafae58e078f44b9bc3fc977f92c4c43. Latest consensus state: current_term: 2 leader_uuid: "bafae58e078f44b9bc3fc977f92c4c43" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:43.914068 19495 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:43.914685 19497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:43.918037 19497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:43.918831 19497 catalog_manager.cc:1259] Loaded cluster ID: eea07e915f3a4c728db042c3d44580c3
I20260929 02:06:43.918972 19497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:43.920066 19497 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:43.920840 19497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Loaded TSK: 0
I20260929 02:06:43.921756 19497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:06:43.935362 18948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:43.942077 18948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:43.945889 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.945568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56934)
0929 02:06:43.945697 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.945778 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 02:06:43.945935 18768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [term 2 FOLLOWER]: Refusing update from remote peer bafae58e078f44b9bc3fc977f92c4c43: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 02:06:43.946630 19495 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 02:06:43.947175 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.946681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56944)
0929 02:06:43.946873 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.947012 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:06:43.947345 18933 consensus_peers.cc:487] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 -> Peer 6c40684958ea42ad8974f33559cbcfe8 (127.11.182.254:44915): Couldn't send request to peer 6c40684958ea42ad8974f33559cbcfe8. Status: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:06:43.949306 19486 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bafae58e078f44b9bc3fc977f92c4c43. Latest consensus state: current_term: 2 leader_uuid: "bafae58e078f44b9bc3fc977f92c4c43" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:43.951370 19485 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bafae58e078f44b9bc3fc977f92c4c43" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:43.951687 19485 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:43.953828 19486 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:06:43.959978 19486 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bafae58e078f44b9bc3fc977f92c4c43" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c40684958ea42ad8974f33559cbcfe8" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 44915 } } peers { permanent_uuid: "fbda9dd051364eadbc87bc57ccd4289a" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 39003 } attrs { promote: false } } peers { permanent_uuid: "bafae58e078f44b9bc3fc977f92c4c43" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 41129 } attrs { promote: false } } }
I20260929 02:06:43.960254 19486 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbda9dd051364eadbc87bc57ccd4289a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:06:43.980425 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.965238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.11.182.194:34729)
0929 02:06:43.973962 (+ 8724us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:43.973967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:43.973983 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:06:43.974036 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:43.974040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:43.974093 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:43.975021 (+ 928us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:43.975205 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.975935 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.978256 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.979403 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.979577 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.979652 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:43.979992 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 02:06:43.980137 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8628,"thread_start_us":8536,"threads_started":1}
I20260929 02:06:43.980526 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.965112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:34729 (local address 127.0.0.1:45826)
0929 02:06:43.965870 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.965891 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:43.966016 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:43.974296 (+ 8280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:43.974303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:43.974331 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:43.974784 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.974797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.976096 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.976103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.978131 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.978141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.979637 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.979643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.979813 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.979850 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:06:43.979954 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 02:06:43.980082 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":50,"thread_start_us":738,"threads_started":1}
I20260929 02:06:43.981359 19211 tablet_service.cc:1467] Processing CreateTablet for tablet 34f5616bca4e422b8824da4d9235e1f5 (DEFAULT_TABLE table=test-table [id=5d2c0d80aff3447484b61037027f8361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:43.981823 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f5616bca4e422b8824da4d9235e1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:43.982614 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.964246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:41601 (local address 127.0.0.1:43862)
0929 02:06:43.964508 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.964527 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:43.964628 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:43.976721 (+ 12093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:43.976726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:43.976759 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:43.976976 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.976989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.978345 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.978351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.980069 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.980080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.981931 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.981936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.982081 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.982119 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:06:43.982190 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 02:06:43.982330 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 02:06:43.983532 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.964414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.11.182.195:41601)
0929 02:06:43.976409 (+ 11995us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:43.976416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:43.976434 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:06:43.976496 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:43.976502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:43.976567 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:43.976674 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:43.977149 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.978191 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.980257 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.981757 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.982970 (+ 1213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.983042 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:43.983089 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 02:06:43.983220 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11875,"thread_start_us":11781,"threads_started":1}
I20260929 02:06:43.984755 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.964756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:35255 (local address 127.0.0.1:43856)
0929 02:06:43.964928 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:43.964947 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:43.965063 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:43.976937 (+ 11874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:43.976941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:43.976965 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:43.977511 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.977521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.978830 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.978835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.980949 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:43.980963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.984030 (+ 3067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.984036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:43.984202 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.984224 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:06:43.984333 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 02:06:43.984458 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":97}
I20260929 02:06:43.984778 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.965703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.11.182.193:35255)
0929 02:06:43.976627 (+ 10924us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:43.976650 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:43.976671 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:06:43.976735 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:43.976739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:43.976808 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:43.976918 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:43.977683 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.978709 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.982961 (+ 4252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:43.983840 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:43.984028 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:43.984102 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:43.984356 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 02:06:43.984481 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10808,"thread_start_us":11313,"threads_started":1}
I20260929 02:06:43.985407 19063 tablet_service.cc:1467] Processing CreateTablet for tablet 34f5616bca4e422b8824da4d9235e1f5 (DEFAULT_TABLE table=test-table [id=5d2c0d80aff3447484b61037027f8361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:43.985798 19063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f5616bca4e422b8824da4d9235e1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:43.987270 19508 tablet_bootstrap.cc:492] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: Bootstrap starting.
I20260929 02:06:43.987522 19352 tablet_service.cc:1467] Processing CreateTablet for tablet 34f5616bca4e422b8824da4d9235e1f5 (DEFAULT_TABLE table=test-table [id=5d2c0d80aff3447484b61037027f8361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:43.987933 19352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f5616bca4e422b8824da4d9235e1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:43.989037 19508 tablet_bootstrap.cc:654] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:43.991463 19508 tablet_bootstrap.cc:492] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: No bootstrap required, opened a new log
I20260929 02:06:43.991605 19508 ts_tablet_manager.cc:1397] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:43.992383 19508 raft_consensus.cc:348] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } }
I20260929 02:06:43.992650 19508 raft_consensus.cc:374] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:43.992730 19508 raft_consensus.cc:729] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed372e6334f4cf58de43c740cbddf2f, State: Initialized, Role: FOLLOWER
I20260929 02:06:43.992857 19510 tablet_bootstrap.cc:492] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: Bootstrap starting.
I20260929 02:06:43.992965 19508 consensus_queue.cc:260] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } }
I20260929 02:06:43.993880 19508 ts_tablet_manager.cc:1428] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 02:06:43.994609 19510 tablet_bootstrap.cc:654] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:43.997496 19276 heartbeater.cc:503] Master 127.11.182.252:41129 was elected leader, sending a full tablet report...
I20260929 02:06:44.003294 19512 tablet_bootstrap.cc:492] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: Bootstrap starting.
I20260929 02:06:44.004115 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:43.994527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.194:36667)
0929 02:06:44.003527 (+ 9000us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.003746 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8719,"thread_start_us":8521,"threads_started":1}
I20260929 02:06:44.005345 19512 tablet_bootstrap.cc:654] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:44.005848 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.005264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.194:51899)
0929 02:06:44.005532 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.005658 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 02:06:44.008180 19510 tablet_bootstrap.cc:492] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: No bootstrap required, opened a new log
I20260929 02:06:44.008181 19512 tablet_bootstrap.cc:492] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: No bootstrap required, opened a new log
I20260929 02:06:44.008338 19510 ts_tablet_manager.cc:1397] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260929 02:06:44.008335 19512 ts_tablet_manager.cc:1397] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 02:06:44.009179 19512 raft_consensus.cc:348] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } }
I20260929 02:06:44.009179 19510 raft_consensus.cc:348] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } }
I20260929 02:06:44.009423 19512 raft_consensus.cc:374] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:44.009423 19510 raft_consensus.cc:374] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:44.009526 19510 raft_consensus.cc:729] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c9eede04a541b1b1f2dd6b37a882a1, State: Initialized, Role: FOLLOWER
I20260929 02:06:44.009526 19512 raft_consensus.cc:729] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81b0df2be8b4d5cb7b042ee400726d2, State: Initialized, Role: FOLLOWER
I20260929 02:06:44.009795 19510 consensus_queue.cc:260] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } }
I20260929 02:06:44.009789 19512 consensus_queue.cc:260] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } }
I20260929 02:06:44.010741 19510 ts_tablet_manager.cc:1428] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:06:44.012156 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.010855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.193:41723)
0929 02:06:44.011572 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.011767 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":523,"threads_started":1}
I20260929 02:06:44.012830 19128 heartbeater.cc:503] Master 127.11.182.252:41129 was elected leader, sending a full tablet report...
I20260929 02:06:44.013487 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.012975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.193:45793)
0929 02:06:44.013176 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.013312 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 02:06:44.014112 19512 ts_tablet_manager.cc:1428] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260929 02:06:44.016263 19417 heartbeater.cc:503] Master 127.11.182.252:41129 was elected leader, sending a full tablet report...
I20260929 02:06:44.016541 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.014811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.195:55019)
0929 02:06:44.015775 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.016161 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":763,"threads_started":1}
I20260929 02:06:44.017709 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.017183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.195:53877)
0929 02:06:44.017484 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.017574 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260929 02:06:44.128330 19516 raft_consensus.cc:482] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:44.128584 19516 raft_consensus.cc:504] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } }
I20260929 02:06:44.129232 19516 leader_election.cc:288] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27c9eede04a541b1b1f2dd6b37a882a1 (127.11.182.193:35255), 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729)
I20260929 02:06:44.130009 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f5616bca4e422b8824da4d9235e1f5" candidate_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed372e6334f4cf58de43c740cbddf2f" is_pre_election: true
I20260929 02:06:44.130009 19083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f5616bca4e422b8824da4d9235e1f5" candidate_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" is_pre_election: true
I20260929 02:06:44.130285 19231 raft_consensus.cc:2454] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81b0df2be8b4d5cb7b042ee400726d2 in term 0.
I20260929 02:06:44.130285 19083 raft_consensus.cc:2454] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81b0df2be8b4d5cb7b042ee400726d2 in term 0.
I20260929 02:06:44.130769 19308 leader_election.cc:302] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ed372e6334f4cf58de43c740cbddf2f, b81b0df2be8b4d5cb7b042ee400726d2; no voters:
I20260929 02:06:44.131102 19516 raft_consensus.cc:2781] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:44.131227 19516 raft_consensus.cc:482] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:44.131323 19516 raft_consensus.cc:3037] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:44.133294 19516 raft_consensus.cc:504] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } }
I20260929 02:06:44.133819 19516 leader_election.cc:288] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [CANDIDATE]: Term 1 election: Requested vote from peers 27c9eede04a541b1b1f2dd6b37a882a1 (127.11.182.193:35255), 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729)
I20260929 02:06:44.134316 19083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f5616bca4e422b8824da4d9235e1f5" candidate_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c9eede04a541b1b1f2dd6b37a882a1"
I20260929 02:06:44.134485 19083 raft_consensus.cc:3037] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:44.134465 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f5616bca4e422b8824da4d9235e1f5" candidate_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed372e6334f4cf58de43c740cbddf2f"
I20260929 02:06:44.134765 19231 raft_consensus.cc:3037] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:44.136801 19231 raft_consensus.cc:2454] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81b0df2be8b4d5cb7b042ee400726d2 in term 1.
I20260929 02:06:44.136806 19083 raft_consensus.cc:2454] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81b0df2be8b4d5cb7b042ee400726d2 in term 1.
I20260929 02:06:44.137249 19308 leader_election.cc:302] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27c9eede04a541b1b1f2dd6b37a882a1, b81b0df2be8b4d5cb7b042ee400726d2; no voters:
I20260929 02:06:44.137588 19516 raft_consensus.cc:2781] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:44.138055 19516 raft_consensus.cc:686] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 LEADER]: Becoming Leader. State: Replica: b81b0df2be8b4d5cb7b042ee400726d2, State: Running, Role: LEADER
I20260929 02:06:44.138350 19516 consensus_queue.cc:237] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } }
I20260929 02:06:44.139667 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.139074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.195:60597)
0929 02:06:44.139279 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.139435 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:06:44.141251 18948 catalog_manager.cc:5495] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b81b0df2be8b4d5cb7b042ee400726d2 (127.11.182.195). New cstate: current_term: 1 leader_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81b0df2be8b4d5cb7b042ee400726d2" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 } health_report { overall_health: UNKNOWN } } }
I20260929 02:06:44.168721 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.163622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.11.182.195:41601)
0929 02:06:44.163866 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:44.163873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:44.163892 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:06:44.164094 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:44.164098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:44.164161 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:44.164270 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:44.164637 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:44.165485 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:44.166957 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:44.167430 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:44.167568 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:44.167978 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:44.168106 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:44.168203 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:44.168355 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 02:06:44.168432 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 02:06:44.171826 19083 raft_consensus.cc:1261] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 1 FOLLOWER]: Refusing update from remote peer b81b0df2be8b4d5cb7b042ee400726d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:44.171959 19231 raft_consensus.cc:1261] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [term 1 FOLLOWER]: Refusing update from remote peer b81b0df2be8b4d5cb7b042ee400726d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:44.172436 19520 consensus_queue.cc:1035] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27c9eede04a541b1b1f2dd6b37a882a1" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:44.172791 19516 consensus_queue.cc:1035] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ed372e6334f4cf58de43c740cbddf2f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:44.175259 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.174598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.194:44025)
0929 02:06:44.174988 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.175103 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277}
I20260929 02:06:44.176398 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.175816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.193:57651)
0929 02:06:44.176072 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.176186 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 02:06:44.180822 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.180301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.195:42719)
0929 02:06:44.180575 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.180677 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260929 02:06:44.181512 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.179921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.193:47211)
0929 02:06:44.181240 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.181350 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1199}
I20260929 02:06:44.182626 19526 mvcc.cc:205] Tried to move back new op lower bound from 7334492586681749504 to 7334492586553229312. Current Snapshot: MvccSnapshot[applied={T|T < 7334492586681749504 or (T in {7334492586681749504})}]
I20260929 02:06:44.187558 18948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:44.187853 18948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:44.191550 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.190105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.195:56397)
0929 02:06:44.191237 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.191344 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1013}
I20260929 02:06:44.192692 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.192074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.11.182.194:37409)
0929 02:06:44.192409 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.192535 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260929 02:06:44.196326 18948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43: Sending DeleteTablet for 3 replicas of tablet 34f5616bca4e422b8824da4d9235e1f5
I20260929 02:06:44.197338 19352 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f5616bca4e422b8824da4d9235e1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:44 UTC) from {username='oryx', principal='oryx/127.11.182.252@KRBTEST.COM'} at 127.0.0.1:43862
I20260929 02:06:44.197608 19063 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f5616bca4e422b8824da4d9235e1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:44 UTC) from {username='oryx', principal='oryx/127.11.182.252@KRBTEST.COM'} at 127.0.0.1:43856
I20260929 02:06:44.197613 19211 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f5616bca4e422b8824da4d9235e1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:44 UTC) from {username='oryx', principal='oryx/127.11.182.252@KRBTEST.COM'} at 127.0.0.1:45826
I20260929 02:06:44.198103 19532 tablet_replica.cc:330] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: stopping tablet replica
I20260929 02:06:44.198313 19532 raft_consensus.cc:2229] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:44.198772 19532 raft_consensus.cc:2258] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:44.199995 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 18934
I20260929 02:06:44.200019 19532 ts_tablet_manager.cc:1905] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:44.204057 19533 tablet_replica.cc:330] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: stopping tablet replica
I20260929 02:06:44.207096 19533 raft_consensus.cc:2229] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:44.207578 19533 raft_consensus.cc:2258] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:44.207836 19532 ts_tablet_manager.cc:1918] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:44.207983 19532 log.cc:1196] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/34f5616bca4e422b8824da4d9235e1f5
I20260929 02:06:44.208468 19532 ts_tablet_manager.cc:1939] T 34f5616bca4e422b8824da4d9235e1f5 P b81b0df2be8b4d5cb7b042ee400726d2: Deleting consensus metadata
I20260929 02:06:44.208967 19533 ts_tablet_manager.cc:1905] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:44.209725 18936 catalog_manager.cc:4841] TS b81b0df2be8b4d5cb7b042ee400726d2 (127.11.182.195:41601): tablet 34f5616bca4e422b8824da4d9235e1f5 (table test-table [id=5d2c0d80aff3447484b61037027f8361]) successfully deleted
I20260929 02:06:44.211133 19534 tablet_replica.cc:330] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: stopping tablet replica
I20260929 02:06:44.211356 19534 raft_consensus.cc:2229] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:44.211668 19534 raft_consensus.cc:2258] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:44.212572 19534 ts_tablet_manager.cc:1905] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:44.216032 19533 ts_tablet_manager.cc:1918] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:44.216205 19533 log.cc:1196] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/34f5616bca4e422b8824da4d9235e1f5
I20260929 02:06:44.216684 19533 ts_tablet_manager.cc:1939] T 34f5616bca4e422b8824da4d9235e1f5 P 2ed372e6334f4cf58de43c740cbddf2f: Deleting consensus metadata
I20260929 02:06:44.217796 18936 catalog_manager.cc:4841] TS 2ed372e6334f4cf58de43c740cbddf2f (127.11.182.194:34729): tablet 34f5616bca4e422b8824da4d9235e1f5 (table test-table [id=5d2c0d80aff3447484b61037027f8361]) successfully deleted
I20260929 02:06:44.218552 19534 ts_tablet_manager.cc:1918] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:44.218683 19534 log.cc:1196] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/34f5616bca4e422b8824da4d9235e1f5
I20260929 02:06:44.219146 19534 ts_tablet_manager.cc:1939] T 34f5616bca4e422b8824da4d9235e1f5 P 27c9eede04a541b1b1f2dd6b37a882a1: Deleting consensus metadata
I20260929 02:06:44.220271 18936 catalog_manager.cc:4841] TS 27c9eede04a541b1b1f2dd6b37a882a1 (127.11.182.193:35255): tablet 34f5616bca4e422b8824da4d9235e1f5 (table test-table [id=5d2c0d80aff3447484b61037027f8361]) successfully deleted
I20260929 02:06:44.222913 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.201324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.11.182.193:35255)
0929 02:06:44.201597 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:44.201605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:44.203393 (+ 1788us) server_negotiation.cc:407] Connection header received
0929 02:06:44.203670 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:44.203679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:44.203787 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:44.203890 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:44.206923 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:44.208103 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:44.210801 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:44.211700 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:44.211844 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:44.215851 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:44.215875 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:44.215902 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:44.215976 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:44.220087 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:44.220850 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:44.220859 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:44.220863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:44.220989 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:44.221366 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:44.221371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:44.221378 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:44.221720 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:44.222002 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:44.222310 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 02:06:44.222519 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:06:44.223610 19123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:44.317646 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 18934
I20260929 02:06:44.334638 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 19140
I20260929 02:06:44.335927 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.335567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56960)
0929 02:06:44.335713 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.335795 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 02:06:44.336925 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.336580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:44915 (local address 127.0.0.1:56966)
0929 02:06:44.336711 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.336801 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:06:44.348066 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.335488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.11.182.194:34729)
0929 02:06:44.335677 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:44.335685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:44.335725 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:06:44.335956 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:44.335961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:44.336061 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:44.336194 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:44.337635 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:44.338346 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:44.340121 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:44.340542 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:44.340725 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:44.343212 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:44.343225 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:44.343252 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:44.343281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:44.345802 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:44.346409 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:44.346415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:44.346418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:44.346523 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:44.346811 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:44.346814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:44.346816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:44.347168 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:44.347396 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:44.347567 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 02:06:44.347727 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:06:44.348714 19271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:44.440812 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 19140
I20260929 02:06:44.463339 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 19289
I20260929 02:06:44.477907 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.464113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.11.182.195:41601)
0929 02:06:44.464328 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:44.464333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:44.464353 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:06:44.464516 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:44.464521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:44.464602 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:44.464706 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:44.466133 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:44.467188 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:44.468664 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:44.469087 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:44.469222 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:44.472219 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:44.472233 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:44.472251 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:44.472281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:44.475571 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:44.476187 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:44.476198 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:44.476204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:44.476341 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:44.476640 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:44.476643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:44.476645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:44.476983 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:44.477212 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:44.477486 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 02:06:44.477629 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:06:44.478540 19412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:44.597179 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 19289
I20260929 02:06:44.618613 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 18622
I20260929 02:06:44.619791 18788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:44.732461 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 18622
W20260929 02:06:44.753083 18933 connection.cc:535] client connection to 127.11.182.253:39003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:06:44.753357 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 18722
I20260929 02:06:44.754308 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:44.753828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:39003 (local address 127.0.0.1:40468)
0929 02:06:44.754015 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:44.754107 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 02:06:44.754384 18979 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 02:06:44.754484 18933 consensus_peers.cc:487] T 00000000000000000000000000000000 P bafae58e078f44b9bc3fc977f92c4c43 -> Peer fbda9dd051364eadbc87bc57ccd4289a (127.11.182.253:39003): Couldn't send request to peer fbda9dd051364eadbc87bc57ccd4289a. Status: Network error: Client connection negotiation failed: client connection to 127.11.182.253:39003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:06:44.872018 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 18722
2026-09-29T02:06:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9549 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:44 dist-test-slave-q06p krb5kdc[19543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:44 dist-test-slave-q06p krb5kdc[19543](info): set up 2 sockets
Sep 29 02:06:44 dist-test-slave-q06p krb5kdc[19543](info): commencing operation
krb5kdc: starting...
W20260929 02:06:45.980298 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:46 dist-test-slave-q06p krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.254.keytab.
Entry for principal oryx/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.254.keytab.
I20260929 02:06:46.199097 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:32873
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.254.keytab
--principal=oryx/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:46.520771 19557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:46.521059 19557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:46.521154 19557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:46.528401 19557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:46.528499 19557 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:06:46.528545 19557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:46.528599 19557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:46.528646 19557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:46.528702 19557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:46.538398 19557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:32873
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.254.keytab
--principal=oryx/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:46.539703 19557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:46.540910 19557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:46.553686 19562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:46.553781 19565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:46.553975 19563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:46.554800 19557 server_base.cc:1034] running on GCE node
I20260929 02:06:46.555670 19557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:46.557139 19557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:46.558410 19557 hybrid_clock.cc:648] HybridClock initialized: now 1790647606558327 us; error 52 us; skew 500 ppm
I20260929 02:06:46.558763 19557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:46 dist-test-slave-q06p krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647606, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:46.562016 19557 init.cc:377] Logged in from keytab as oryx/127.11.182.254@KRBTEST.COM (short username oryx)
I20260929 02:06:46.563617 19557 webserver.cc:460] Webserver started at http://127.11.182.254:33985/ using document root <none> and password file <none>
I20260929 02:06:46.564327 19557 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:46.564448 19557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:46.564775 19557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:46.567772 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "02e7babdf14a487d96d7c6428f6c90a6"
format_stamp: "Formatted at 2026-09-29 02:06:46 on dist-test-slave-q06p"
server_key: "0bc089f10e8b46c33377463e654a84be"
server_key_iv: "5c31114b47b40703b1d3150b0345ed9e"
server_key_version: "encryptionkey@0"
I20260929 02:06:46.568588 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "02e7babdf14a487d96d7c6428f6c90a6"
format_stamp: "Formatted at 2026-09-29 02:06:46 on dist-test-slave-q06p"
server_key: "0bc089f10e8b46c33377463e654a84be"
server_key_iv: "5c31114b47b40703b1d3150b0345ed9e"
server_key_version: "encryptionkey@0"
I20260929 02:06:46.573779 19557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:06:46.577375 19572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:46.578748 19557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:06:46.578976 19557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "02e7babdf14a487d96d7c6428f6c90a6"
format_stamp: "Formatted at 2026-09-29 02:06:46 on dist-test-slave-q06p"
server_key: "0bc089f10e8b46c33377463e654a84be"
server_key_iv: "5c31114b47b40703b1d3150b0345ed9e"
server_key_version: "encryptionkey@0"
I20260929 02:06:46.579203 19557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:46.592806 19557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:46.596803 19557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:46.597060 19557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:46.619277 19557 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:32873
I20260929 02:06:46.619298 19633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:32873 every 8 connection(s)
I20260929 02:06:46.620622 19557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:46.624470 19634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:46.630095 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 19557
I20260929 02:06:46.630303 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:46.631129 11995 external_mini_cluster.cc:1428] Setting key 21eaa3db24a16ce9195d6c144f60ae94
I20260929 02:06:46.635473 19634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6: Bootstrap starting.
I20260929 02:06:46.639218 19634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:46.640421 19634 log.cc:824] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:46.643520 19634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6: No bootstrap required, opened a new log
Sep 29 02:06:46 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:46.654035 19634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e7babdf14a487d96d7c6428f6c90a6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 32873 } }
I20260929 02:06:46.654482 19634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:46.654597 19634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e7babdf14a487d96d7c6428f6c90a6, State: Initialized, Role: FOLLOWER
I20260929 02:06:46.655300 19634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e7babdf14a487d96d7c6428f6c90a6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 32873 } }
I20260929 02:06:46.655587 19634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:46.655728 19634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:46.655905 19634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:46.656047 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:46.633224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.11.182.254:32873)
0929 02:06:46.634001 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:46.634012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:46.634052 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:06:46.635780 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:46.635803 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:46.636199 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:46.636513 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:46.638364 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:46.639804 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:46.641482 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:46.642183 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:46.642312 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:46.647057 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:46.647077 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:46.647100 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:46.647157 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:46.651288 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:46.652006 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:46.652023 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:46.652032 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:46.652270 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:46.652606 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:46.652611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:46.652614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:46.653407 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:46.653671 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:46.654101 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 02:06:46.654582 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":387,"threads_started":1}
I20260929 02:06:46.658265 19634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e7babdf14a487d96d7c6428f6c90a6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 32873 } }
I20260929 02:06:46.658805 19634 leader_election.cc:302] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02e7babdf14a487d96d7c6428f6c90a6; no voters:
I20260929 02:06:46.659442 19634 leader_election.cc:288] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:46.659534 19639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:46.660264 19639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [term 1 LEADER]: Becoming Leader. State: Replica: 02e7babdf14a487d96d7c6428f6c90a6, State: Running, Role: LEADER
I20260929 02:06:46.660823 19639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e7babdf14a487d96d7c6428f6c90a6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 32873 } }
I20260929 02:06:46.661512 19634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:46.663475 19641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02e7babdf14a487d96d7c6428f6c90a6. Latest consensus state: current_term: 1 leader_uuid: "02e7babdf14a487d96d7c6428f6c90a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e7babdf14a487d96d7c6428f6c90a6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 32873 } } }
I20260929 02:06:46.663851 19641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:46.664798 19644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:46.664621 19640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02e7babdf14a487d96d7c6428f6c90a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e7babdf14a487d96d7c6428f6c90a6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 32873 } } }
I20260929 02:06:46.665005 19640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:46.675107 19644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:46.685151 19644 catalog_manager.cc:1347] Generated new cluster ID: 78b50ccf06ce4f1bacefe0bc2ca73ead
I20260929 02:06:46.685312 19644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:46.695401 19644 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:46.696762 19644 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:46.703740 19644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6: Generated new TSK 0
I20260929 02:06:46.704636 19644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab.
Entry for principal oryx/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab.
I20260929 02:06:46.771479 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:32873
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab
--principal=oryx/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:47.106805 19660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:47.107091 19660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:47.107187 19660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:47.114305 19660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:47.114415 19660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:47.114498 19660 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:06:47.114578 19660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:47.124401 19660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab
--principal=oryx/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.182.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:47.125617 19660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:47.126945 19660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:47.139765 19665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:47.139789 19666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:47.139935 19668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:47.140599 19660 server_base.cc:1034] running on GCE node
I20260929 02:06:47.141265 19660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:47.142146 19660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:47.143383 19660 hybrid_clock.cc:648] HybridClock initialized: now 1790647607143308 us; error 54 us; skew 500 ppm
I20260929 02:06:47.143678 19660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:47 dist-test-slave-q06p krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647607, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:47.146667 19660 init.cc:377] Logged in from keytab as oryx/127.11.182.193@KRBTEST.COM (short username oryx)
I20260929 02:06:47.148186 19660 webserver.cc:460] Webserver started at http://127.11.182.193:41447/ using document root <none> and password file <none>
I20260929 02:06:47.148878 19660 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:47.149009 19660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:47.149322 19660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:47.152171 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "82a7fd9b71b74a05b45760ca43b7b1cb"
format_stamp: "Formatted at 2026-09-29 02:06:47 on dist-test-slave-q06p"
server_key: "8fac0df0b7e631eab4266472d721e174"
server_key_iv: "37905906bb3dad542cbbf0e8e6d967ab"
server_key_version: "encryptionkey@0"
I20260929 02:06:47.152961 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "82a7fd9b71b74a05b45760ca43b7b1cb"
format_stamp: "Formatted at 2026-09-29 02:06:47 on dist-test-slave-q06p"
server_key: "8fac0df0b7e631eab4266472d721e174"
server_key_iv: "37905906bb3dad542cbbf0e8e6d967ab"
server_key_version: "encryptionkey@0"
I20260929 02:06:47.157930 19660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:06:47.161235 19675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:47.162663 19660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:47.162890 19660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "82a7fd9b71b74a05b45760ca43b7b1cb"
format_stamp: "Formatted at 2026-09-29 02:06:47 on dist-test-slave-q06p"
server_key: "8fac0df0b7e631eab4266472d721e174"
server_key_iv: "37905906bb3dad542cbbf0e8e6d967ab"
server_key_version: "encryptionkey@0"
I20260929 02:06:47.163112 19660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:47.174638 19660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:47.178776 19660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:47.179060 19660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:47.181789 19660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:47.181912 19660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:47.182080 19660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:47.182163 19660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:06:47 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647607, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:47.206182 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:47.182068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:52211 (local address 127.11.182.254:32873)
0929 02:06:47.182574 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:47.182584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:47.184084 (+ 1500us) server_negotiation.cc:407] Connection header received
0929 02:06:47.186255 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:47.186261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:47.186353 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:47.186476 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:47.189109 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.190279 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.192906 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.193488 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.193616 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:47.198815 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:47.198825 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:47.198828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:47.198914 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:47.202350 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:47.203364 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:47.203369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:47.203373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:47.203506 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:47.204175 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:47.204180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:47.204183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:47.204473 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:47.204627 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:47.205583 (+ 956us) server_negotiation.cc:299] Negotiation successful
0929 02:06:47.205811 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260929 02:06:47.207769 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:47.182916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:32873 (local address 127.11.182.193:52211)
0929 02:06:47.183910 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:47.183967 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:47.185940 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:47.186931 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:47.186946 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:47.187840 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:47.188856 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:47.188890 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.190486 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.190494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:47.192755 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:47.192766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.193735 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.193750 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:47.194005 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:47.194975 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:47.194995 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:47.198478 (+ 3483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:47.202490 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:47.202500 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:47.202519 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:47.203168 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:47.203673 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:47.203682 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:47.203686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:47.204013 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:47.204609 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:47.204616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:47.205125 (+ 509us) client_negotiation.cc:769] Sending connection context
0929 02:06:47.205498 (+ 373us) client_negotiation.cc:240] Negotiation successful
0929 02:06:47.206093 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":324,"threads_started":1}
I20260929 02:06:47.224953 19660 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:46259
I20260929 02:06:47.224979 19795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:46259 every 8 connection(s)
I20260929 02:06:47.226181 19660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:47.228626 19796 heartbeater.cc:347] Connected to a master server at 127.11.182.254:32873
I20260929 02:06:47.228873 19796 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:47.229544 19796 heartbeater.cc:511] Master 127.11.182.254:32873 requested a full tablet report, sending...
I20260929 02:06:47.231230 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:47.216393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:58769 (local address 127.11.182.254:32873)
0929 02:06:47.216610 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:47.216619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:47.217668 (+ 1049us) server_negotiation.cc:407] Connection header received
0929 02:06:47.217869 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:47.217873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:47.218014 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:47.218152 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:47.219821 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.220600 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.223026 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.223424 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.223553 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:47.225690 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:47.225696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:47.225699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:47.225728 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:47.228612 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:47.229292 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:47.229301 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:47.229305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:47.229428 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:47.229797 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:47.229803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:47.229806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:47.230058 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:47.230212 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:47.230605 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 02:06:47.230812 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:06:47.231272 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:47.215939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:32873 (local address 127.11.182.193:58769)
0929 02:06:47.217523 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:47.217550 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:47.217742 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:47.218184 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:47.218189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:47.219009 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:47.219610 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:47.219630 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.220808 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.220814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:47.222823 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:47.222835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.223636 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.223644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:47.223820 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:47.224564 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:47.224605 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:47.225498 (+ 893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:47.228733 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:47.228739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:47.228743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:47.229179 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:47.229537 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:47.229542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:47.229546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:47.229705 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:47.230179 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:47.230184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:47.230361 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 02:06:47.230497 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 02:06:47.230814 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"thread_start_us":1045,"threads_started":1}
I20260929 02:06:47.231624 19589 ts_manager.cc:194] Registered new tserver with Master: 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259)
I20260929 02:06:47.233081 19589 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:52211
I20260929 02:06:47.234928 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 19660
I20260929 02:06:47.235129 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:47.235900 11995 external_mini_cluster.cc:1428] Setting key a58627da9dcc1bc09e0c4e58fd0bcb5e
WARNING: no policy specified for oryx/127.11.182.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab.
Entry for principal oryx/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab.
I20260929 02:06:47.281868 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:32873
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab
--principal=oryx/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:47.601220 19800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:47.601714 19800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:47.601887 19800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:47.610442 19800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:47.610586 19800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:47.610740 19800 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:06:47.610900 19800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:47.622772 19800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab
--principal=oryx/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.182.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:47.624301 19800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:47.626014 19800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:47.641933 19806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:47.642061 19805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:47.642078 19808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:47.643092 19800 server_base.cc:1034] running on GCE node
I20260929 02:06:47.643813 19800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:47.644790 19800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:47.646050 19800 hybrid_clock.cc:648] HybridClock initialized: now 1790647607645984 us; error 40 us; skew 500 ppm
I20260929 02:06:47.646382 19800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:47 dist-test-slave-q06p krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647607, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:47.649462 19800 init.cc:377] Logged in from keytab as oryx/127.11.182.194@KRBTEST.COM (short username oryx)
I20260929 02:06:47.650942 19800 webserver.cc:460] Webserver started at http://127.11.182.194:36019/ using document root <none> and password file <none>
I20260929 02:06:47.651626 19800 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:47.651762 19800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:47.652071 19800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:47.654950 19800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "52c0b5d64a2d46ffba760531e8ce4350"
format_stamp: "Formatted at 2026-09-29 02:06:47 on dist-test-slave-q06p"
server_key: "c6250515591920c187d867416244b012"
server_key_iv: "35632a081905578277cfebcf26734deb"
server_key_version: "encryptionkey@0"
I20260929 02:06:47.655718 19800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "52c0b5d64a2d46ffba760531e8ce4350"
format_stamp: "Formatted at 2026-09-29 02:06:47 on dist-test-slave-q06p"
server_key: "c6250515591920c187d867416244b012"
server_key_iv: "35632a081905578277cfebcf26734deb"
server_key_version: "encryptionkey@0"
I20260929 02:06:47.660650 19800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:06:47.664270 19815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:47.665524 19800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 02:06:47.665704 19800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "52c0b5d64a2d46ffba760531e8ce4350"
format_stamp: "Formatted at 2026-09-29 02:06:47 on dist-test-slave-q06p"
server_key: "c6250515591920c187d867416244b012"
server_key_iv: "35632a081905578277cfebcf26734deb"
server_key_version: "encryptionkey@0"
I20260929 02:06:47.665922 19800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:47.691598 19800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:47.695430 19800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:47.695708 19800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:47.698340 19800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:47.698441 19800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:47.698590 19800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:47.698683 19800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:06:47 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647607, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:47.723610 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:47.698804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:47625 (local address 127.11.182.254:32873)
0929 02:06:47.699018 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:47.699028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:47.700503 (+ 1475us) server_negotiation.cc:407] Connection header received
0929 02:06:47.702669 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:47.702677 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:47.702777 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:47.702964 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:47.705121 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.706193 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.709097 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.709697 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.709837 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:47.715769 (+ 5932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:47.715774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:47.715778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:47.715810 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:47.718332 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:47.719264 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:47.719273 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:47.719280 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:47.719463 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:47.719955 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:47.719961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:47.719965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:47.720331 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:47.720497 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:47.723058 (+ 2561us) server_negotiation.cc:299] Negotiation successful
0929 02:06:47.723309 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:06:47.724339 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:47.699546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:32873 (local address 127.11.182.194:47625)
0929 02:06:47.700346 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:47.700390 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:47.702304 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:47.703219 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:47.703232 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:47.703984 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:47.704917 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:47.704948 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.706373 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.706378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:47.708856 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:47.708877 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.709990 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.709999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:47.710377 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:47.711292 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:47.711338 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:47.715491 (+ 4153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:47.718502 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:47.718509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:47.718523 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:47.719080 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:47.719601 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:47.719605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:47.719607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:47.719822 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:47.720508 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:47.720519 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:47.721136 (+ 617us) client_negotiation.cc:769] Sending connection context
0929 02:06:47.721552 (+ 416us) client_negotiation.cc:240] Negotiation successful
0929 02:06:47.722316 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":367,"threads_started":1}
I20260929 02:06:47.744809 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:47.730017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:42633 (local address 127.11.182.254:32873)
0929 02:06:47.730176 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:47.730180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:47.730834 (+ 654us) server_negotiation.cc:407] Connection header received
0929 02:06:47.731063 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:47.731067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:47.731134 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:47.731232 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:47.732192 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.732915 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.734193 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.734607 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.734754 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:47.737645 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:47.737658 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:47.737661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:47.737692 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:47.741752 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:47.742502 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:47.742510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:47.742517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:47.742721 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:47.743160 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:47.743169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:47.743172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:47.743623 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:47.743787 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:47.744234 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 02:06:47.744455 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:06:47.744964 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:47.730152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:32873 (local address 127.11.182.194:42633)
0929 02:06:47.730731 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:47.730749 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:47.730917 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:47.731261 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:47.731266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:47.731725 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:47.732059 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:47.732069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.733045 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.733051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:47.734058 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:47.734068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:47.734791 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:47.734798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:47.735031 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:47.735851 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:47.735883 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:47.737423 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:47.741924 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:47.741962 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:47.741965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:47.742380 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:47.742901 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:47.742908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:47.742911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:47.743025 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:47.743744 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:47.743749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:47.743946 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 02:06:47.744138 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 02:06:47.744488 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":330,"threads_started":1}
I20260929 02:06:47.755642 19800 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:33491
I20260929 02:06:47.755664 19934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:33491 every 8 connection(s)
I20260929 02:06:47.756906 19800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:47.758996 19935 heartbeater.cc:347] Connected to a master server at 127.11.182.254:32873
I20260929 02:06:47.759223 19935 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:47.759876 19935 heartbeater.cc:511] Master 127.11.182.254:32873 requested a full tablet report, sending...
I20260929 02:06:47.761190 19589 ts_manager.cc:194] Registered new tserver with Master: 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194:33491)
I20260929 02:06:47.762115 19589 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:47625
I20260929 02:06:47.762625 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 19800
I20260929 02:06:47.762811 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:47.763288 11995 external_mini_cluster.cc:1428] Setting key ec0f2f3f73330aebadf24d6b486e9a38
WARNING: no policy specified for oryx/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab.
Entry for principal oryx/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab.
I20260929 02:06:47.811192 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:32873
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab
--principal=oryx/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:48.145817 19939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:48.146119 19939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:48.146271 19939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:48.153633 19939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:48.153757 19939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:48.153903 19939 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:06:48.154023 19939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:48.163960 19939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab
--principal=oryx/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.182.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:48.165217 19939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:48.166615 19939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:48.180666 19944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:48.180593 19947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:48.180627 19945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:48.181525 19939 server_base.cc:1034] running on GCE node
I20260929 02:06:48.182206 19939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:48.183302 19939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:48.184554 19939 hybrid_clock.cc:648] HybridClock initialized: now 1790647608184483 us; error 51 us; skew 500 ppm
I20260929 02:06:48.184942 19939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:48 dist-test-slave-q06p krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647608, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:48.188141 19939 init.cc:377] Logged in from keytab as oryx/127.11.182.195@KRBTEST.COM (short username oryx)
I20260929 02:06:48.189613 19939 webserver.cc:460] Webserver started at http://127.11.182.195:45549/ using document root <none> and password file <none>
I20260929 02:06:48.190265 19939 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:48.190349 19939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:48.190613 19939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:48.193598 19939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "4b57f799c569438a865f3a0f8af4c480"
format_stamp: "Formatted at 2026-09-29 02:06:48 on dist-test-slave-q06p"
server_key: "710f12a0fdbcb56a377eb07a9435c07c"
server_key_iv: "78191e9a3bd2b9ddcd1d91b2ab773773"
server_key_version: "encryptionkey@0"
I20260929 02:06:48.194342 19939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "4b57f799c569438a865f3a0f8af4c480"
format_stamp: "Formatted at 2026-09-29 02:06:48 on dist-test-slave-q06p"
server_key: "710f12a0fdbcb56a377eb07a9435c07c"
server_key_iv: "78191e9a3bd2b9ddcd1d91b2ab773773"
server_key_version: "encryptionkey@0"
I20260929 02:06:48.199852 19939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:06:48.203507 19954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:48.204946 19939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:06:48.205081 19939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "4b57f799c569438a865f3a0f8af4c480"
format_stamp: "Formatted at 2026-09-29 02:06:48 on dist-test-slave-q06p"
server_key: "710f12a0fdbcb56a377eb07a9435c07c"
server_key_iv: "78191e9a3bd2b9ddcd1d91b2ab773773"
server_key_version: "encryptionkey@0"
I20260929 02:06:48.205287 19939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:48.225729 19939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:48.231298 19939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:48.231539 19939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:48.234164 19939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:48.234287 19939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:48.234426 19939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:48.234493 19939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:48.236263 19796 heartbeater.cc:503] Master 127.11.182.254:32873 was elected leader, sending a full tablet report...
Sep 29 02:06:48 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647608, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:48.259692 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.235010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:53793 (local address 127.11.182.254:32873)
0929 02:06:48.235215 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.235241 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.237261 (+ 2020us) server_negotiation.cc:407] Connection header received
0929 02:06:48.239694 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.239700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.239782 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.239879 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:48.242139 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.243301 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.246003 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.246726 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.246907 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.251872 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:48.251876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:48.251880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:48.251915 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:48.254556 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:48.255764 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:48.255771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:48.255774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:48.255920 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:48.256536 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:48.256541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:48.256545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:48.256963 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:48.257110 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:48.258772 (+ 1662us) server_negotiation.cc:299] Negotiation successful
0929 02:06:48.259145 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:06:48.260419 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.235756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:32873 (local address 127.11.182.195:53793)
0929 02:06:48.237096 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:48.237153 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:48.239404 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:48.240146 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:48.240154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:48.240943 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:48.241924 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.241954 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.243500 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.243504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.245812 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.245835 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.247106 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.247112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.247307 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.248028 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:48.248059 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:48.251546 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:48.254716 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:48.254730 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:48.254755 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:48.255589 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:48.256048 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:48.256056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:48.256060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:48.256409 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:48.257173 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:48.257180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:48.257753 (+ 573us) client_negotiation.cc:769] Sending connection context
0929 02:06:48.258142 (+ 389us) client_negotiation.cc:240] Negotiation successful
0929 02:06:48.259011 (+ 869us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"thread_start_us":756,"threads_started":1}
I20260929 02:06:48.284868 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.267230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:32873 (local address 127.11.182.195:44531)
0929 02:06:48.268022 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:48.268048 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:48.268235 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:48.268617 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:48.268624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:48.269403 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:48.269891 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.269933 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.271679 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.271689 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.274042 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.274068 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.275019 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.275028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.275322 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.276093 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:48.276126 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:48.277505 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:48.281787 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:48.281795 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:48.281803 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:48.282241 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:48.282782 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:48.282790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:48.282796 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:48.283053 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:48.283879 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:48.283883 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:48.284129 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 02:06:48.284249 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 02:06:48.284513 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":399,"threads_started":1}
I20260929 02:06:48.285254 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.267120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:44531 (local address 127.11.182.254:32873)
0929 02:06:48.267343 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.267352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.268153 (+ 801us) server_negotiation.cc:407] Connection header received
0929 02:06:48.268374 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.268378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.268462 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.268547 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:48.270071 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.271016 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.274234 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.274754 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.274928 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.277766 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:48.277771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:48.277774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:48.277808 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:48.281578 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:48.282384 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:48.282393 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:48.282399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:48.282605 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:48.283175 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:48.283183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:48.283189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:48.283682 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:48.283863 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:48.284433 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 02:06:48.284754 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:06:48.295836 19939 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:44059
I20260929 02:06:48.295867 20073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:44059 every 8 connection(s)
I20260929 02:06:48.297531 19939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:48.299808 20074 heartbeater.cc:347] Connected to a master server at 127.11.182.254:32873
I20260929 02:06:48.300045 20074 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:48.300746 20074 heartbeater.cc:511] Master 127.11.182.254:32873 requested a full tablet report, sending...
I20260929 02:06:48.302249 19589 ts_manager.cc:194] Registered new tserver with Master: 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:48.303213 19589 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:53793
I20260929 02:06:48.303680 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 19939
I20260929 02:06:48.303853 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:48.304425 11995 external_mini_cluster.cc:1428] Setting key 5b25388ad7969f401d549a50be1fea56
I20260929 02:06:48.307369 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:06:48 dist-test-slave-q06p krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647608, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:06:48 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:48 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:06:48.358686 19683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:48.347519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.11.182.254:32873)
0929 02:06:48.347786 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.347793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.348016 (+ 223us) server_negotiation.cc:407] Connection header received
0929 02:06:48.348252 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.348258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.348326 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.348455 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:48.349915 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.350708 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.352051 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.352683 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.352838 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.358382 (+ 5544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41998: BlockingRecv error: recv got EOF from 127.0.0.1:41998 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 29 02:06:48 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647608, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:48.376010 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.361976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.11.182.254:32873)
0929 02:06:48.362139 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.362143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.362645 (+ 502us) server_negotiation.cc:407] Connection header received
0929 02:06:48.362855 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.362880 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.362973 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.363069 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:48.364295 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.365062 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.366337 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.366757 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.366910 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.370595 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:48.370601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:48.370604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:48.370690 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:48.373336 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:48.374190 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:48.374195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:48.374197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:48.374327 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:48.374667 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:48.374670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:48.374672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:48.375054 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:48.375290 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:48.375560 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 02:06:48.375741 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:06:48.380102 19589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:48.387004 19589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:48.427201 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.399872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.11.182.193:46259)
0929 02:06:48.417231 (+ 17359us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.417245 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.417285 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:06:48.417414 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.417425 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.418667 (+ 1242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.418948 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:48.420104 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.421654 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.424445 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.425939 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.426122 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.426324 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:48.426691 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 02:06:48.426904 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17207,"thread_start_us":17096,"threads_started":1}
I20260929 02:06:48.427662 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.399793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:46259 (local address 127.0.0.1:38910)
0929 02:06:48.416281 (+ 16488us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:48.416340 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:48.416611 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:48.419271 (+ 2660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:48.419283 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:48.419338 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:48.419862 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.419881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.421825 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.421832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.424272 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.424287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.426218 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.426225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.426418 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.426472 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 02:06:48.426616 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 02:06:48.426876 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16089,"thread_start_us":17904,"threads_started":1}
I20260929 02:06:48.429263 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.400040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:33491 (local address 127.0.0.1:33030)
0929 02:06:48.420105 (+ 20065us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:48.420126 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:48.420236 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:48.421994 (+ 1758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:48.421999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:48.422036 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:48.422527 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.422540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.424404 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.424409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.426678 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.426691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.428479 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.428485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.428659 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.428693 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:06:48.428798 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:06:48.428923 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19926,"thread_start_us":19770,"threads_started":1}
I20260929 02:06:48.430083 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.401546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.11.182.194:33491)
0929 02:06:48.421383 (+ 19837us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.421391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.421418 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:06:48.421502 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.421510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.421781 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.421984 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:48.422707 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.424227 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.426869 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.428283 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.428451 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.428599 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:48.428858 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 02:06:48.429813 (+ 955us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19685,"thread_start_us":19557,"threads_started":1}
I20260929 02:06:48.430409 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.400616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:44059 (local address 127.0.0.1:57998)
0929 02:06:48.421970 (+ 21354us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:48.421992 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:48.422121 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:48.422837 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:48.422868 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:48.422897 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:48.423294 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.423302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.425137 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.425145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.427030 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.427043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.429715 (+ 2672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.429720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.429831 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.429851 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 02:06:48.429945 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:06:48.430066 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21203,"mutex_wait_us":1,"thread_start_us":21057,"threads_started":1}
I20260929 02:06:48.430507 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.401525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.11.182.195:44059)
0929 02:06:48.420798 (+ 19273us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.420807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.422268 (+ 1461us) server_negotiation.cc:407] Connection header received
0929 02:06:48.422385 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.422394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.422651 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.422883 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:48.423534 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.424994 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.427730 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.429278 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.429466 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.429602 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:48.430005 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 02:06:48.430183 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19094,"thread_start_us":19266,"threads_started":1}
I20260929 02:06:48.433534 19865 tablet_service.cc:1467] Processing CreateTablet for tablet 582bff493d324ebc965e61609d5ec08b (DEFAULT_TABLE table=test-table [id=d0fbe03ac325475b973ff024c2aee609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:48.433980 20004 tablet_service.cc:1467] Processing CreateTablet for tablet 582bff493d324ebc965e61609d5ec08b (DEFAULT_TABLE table=test-table [id=d0fbe03ac325475b973ff024c2aee609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:48.432054 19726 tablet_service.cc:1467] Processing CreateTablet for tablet 582bff493d324ebc965e61609d5ec08b (DEFAULT_TABLE table=test-table [id=d0fbe03ac325475b973ff024c2aee609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:48.434744 19865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582bff493d324ebc965e61609d5ec08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:48.435282 19726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582bff493d324ebc965e61609d5ec08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:48.435293 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582bff493d324ebc965e61609d5ec08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:48.446058 20098 tablet_bootstrap.cc:492] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: Bootstrap starting.
I20260929 02:06:48.449172 20098 tablet_bootstrap.cc:654] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:48.449761 20099 tablet_bootstrap.cc:492] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: Bootstrap starting.
I20260929 02:06:48.451874 20098 log.cc:824] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:48.453382 20099 tablet_bootstrap.cc:654] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:48.453810 20100 tablet_bootstrap.cc:492] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: Bootstrap starting.
I20260929 02:06:48.454589 20099 log.cc:824] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:48.455154 20098 tablet_bootstrap.cc:492] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: No bootstrap required, opened a new log
I20260929 02:06:48.455446 20098 ts_tablet_manager.cc:1397] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 02:06:48.456840 20100 tablet_bootstrap.cc:654] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:48.457710 20099 tablet_bootstrap.cc:492] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: No bootstrap required, opened a new log
I20260929 02:06:48.457830 20100 log.cc:824] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:48.457980 20099 ts_tablet_manager.cc:1397] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 02:06:48.460846 20100 tablet_bootstrap.cc:492] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: No bootstrap required, opened a new log
I20260929 02:06:48.461120 20100 ts_tablet_manager.cc:1397] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:06:48.466825 20098 raft_consensus.cc:348] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.467382 20098 raft_consensus.cc:374] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:48.467482 20098 raft_consensus.cc:729] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c0b5d64a2d46ffba760531e8ce4350, State: Initialized, Role: FOLLOWER
I20260929 02:06:48.468204 20098 consensus_queue.cc:260] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.468760 20099 raft_consensus.cc:348] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.469295 20099 raft_consensus.cc:374] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:48.469091 20100 raft_consensus.cc:348] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.469393 20099 raft_consensus.cc:729] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b57f799c569438a865f3a0f8af4c480, State: Initialized, Role: FOLLOWER
I20260929 02:06:48.469477 19935 heartbeater.cc:503] Master 127.11.182.254:32873 was elected leader, sending a full tablet report...
I20260929 02:06:48.469491 20100 raft_consensus.cc:374] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:48.469591 20100 raft_consensus.cc:729] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a7fd9b71b74a05b45760ca43b7b1cb, State: Initialized, Role: FOLLOWER
I20260929 02:06:48.470178 20098 ts_tablet_manager.cc:1428] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 02:06:48.470118 20099 consensus_queue.cc:260] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.470237 20100 consensus_queue.cc:260] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.473489 20100 ts_tablet_manager.cc:1428] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260929 02:06:48.474067 20074 heartbeater.cc:503] Master 127.11.182.254:32873 was elected leader, sending a full tablet report...
I20260929 02:06:48.475792 20099 ts_tablet_manager.cc:1428] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
W20260929 02:06:48.478976 19797 tablet.cc:2369] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:06:48.510380 19936 tablet.cc:2369] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:06:48.549274 20075 tablet.cc:2369] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:06:48.677340 20104 raft_consensus.cc:482] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:48.677650 20104 raft_consensus.cc:504] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.679351 20104 leader_election.cc:288] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:48.686479 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.679744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:59281 (local address 127.11.182.193:46259)
0929 02:06:48.680009 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.680018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.680729 (+ 711us) server_negotiation.cc:407] Connection header received
0929 02:06:48.680886 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.680894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.680976 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.681063 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:48.681925 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.682612 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.684752 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.685518 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.685712 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.685796 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:48.686113 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 02:06:48.686238 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:06:48.686476 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.679827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:34725 (local address 127.11.182.195:44059)
0929 02:06:48.680035 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.680044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.680944 (+ 900us) server_negotiation.cc:407] Connection header received
0929 02:06:48.681084 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.681088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.681143 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.681210 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:48.681738 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.682581 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.684729 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.685487 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.685680 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.685760 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:48.686087 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 02:06:48.686219 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:06:48.686676 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.679814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:44059 (local address 127.11.182.194:34725)
0929 02:06:48.680839 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:48.680855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:48.680974 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:48.681253 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:48.681257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:48.681274 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:48.681570 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.681579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.682706 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.682712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.684579 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.684591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.685718 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.685723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.685894 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.685955 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 02:06:48.686063 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 02:06:48.686195 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":86,"thread_start_us":441,"threads_started":1}
I20260929 02:06:48.687045 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.679544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:46259 (local address 127.11.182.194:59281)
0929 02:06:48.680607 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:48.680629 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:48.680750 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:48.681360 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:48.681363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:48.681394 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:48.681783 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.681791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.682765 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.682770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.684571 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:48.684583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.685751 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.685757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:48.685956 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.685978 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:06:48.686084 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:06:48.686208 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":66,"thread_start_us":735,"threads_started":1}
I20260929 02:06:48.687803 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bff493d324ebc965e61609d5ec08b" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480" is_pre_election: true
I20260929 02:06:48.688036 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bff493d324ebc965e61609d5ec08b" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" is_pre_election: true
I20260929 02:06:48.688336 20029 raft_consensus.cc:2454] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 0.
I20260929 02:06:48.688550 19746 raft_consensus.cc:2454] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 0.
I20260929 02:06:48.689037 19819 leader_election.cc:302] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b57f799c569438a865f3a0f8af4c480, 52c0b5d64a2d46ffba760531e8ce4350; no voters:
I20260929 02:06:48.689440 20104 raft_consensus.cc:2781] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:48.689589 20104 raft_consensus.cc:482] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:48.689690 20104 raft_consensus.cc:3037] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:48.692018 20104 raft_consensus.cc:504] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.692695 20104 leader_election.cc:288] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 election: Requested vote from peers 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:48.693171 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bff493d324ebc965e61609d5ec08b" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb"
I20260929 02:06:48.693248 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bff493d324ebc965e61609d5ec08b" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480"
I20260929 02:06:48.693373 19746 raft_consensus.cc:3037] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:48.693490 20029 raft_consensus.cc:3037] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:48.695737 19746 raft_consensus.cc:2454] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 1.
I20260929 02:06:48.695806 20029 raft_consensus.cc:2454] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 1.
I20260929 02:06:48.696208 19817 leader_election.cc:302] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c0b5d64a2d46ffba760531e8ce4350, 82a7fd9b71b74a05b45760ca43b7b1cb; no voters:
I20260929 02:06:48.696561 20104 raft_consensus.cc:2781] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:48.696995 20104 raft_consensus.cc:686] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 LEADER]: Becoming Leader. State: Replica: 52c0b5d64a2d46ffba760531e8ce4350, State: Running, Role: LEADER
I20260929 02:06:48.697537 20104 consensus_queue.cc:237] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.702258 19589 catalog_manager.cc:5495] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194). New cstate: current_term: 1 leader_uuid: "52c0b5d64a2d46ffba760531e8ce4350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } health_report { overall_health: UNKNOWN } } }
I20260929 02:06:48.730093 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:48.724731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.11.182.194:33491)
0929 02:06:48.724934 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:48.724939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:48.725017 (+ 78us) server_negotiation.cc:407] Connection header received
0929 02:06:48.725165 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:48.725168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:48.725252 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:48.725384 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:48.725802 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.726529 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.728098 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:48.728504 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:48.728604 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:48.729009 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:48.729573 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:48.729695 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:48.729827 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 02:06:48.729893 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:06:48.738188 19746 raft_consensus.cc:1261] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Refusing update from remote peer 52c0b5d64a2d46ffba760531e8ce4350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:48.738183 20029 raft_consensus.cc:1261] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Refusing update from remote peer 52c0b5d64a2d46ffba760531e8ce4350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:48.739256 20104 consensus_queue.cc:1035] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:48.739560 20109 consensus_queue.cc:1035] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:48.752478 20112 mvcc.cc:205] Tried to move back new op lower bound from 7334492605382062080 to 7334492605229146112. Current Snapshot: MvccSnapshot[applied={T|T < 7334492605382062080}]
I20260929 02:06:48.752797 20113 mvcc.cc:205] Tried to move back new op lower bound from 7334492605382062080 to 7334492605229146112. Current Snapshot: MvccSnapshot[applied={T|T < 7334492605382062080}]
I20260929 02:06:48.769702 19589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:48.769977 19589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:48.774261 19589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6: Sending DeleteTablet for 3 replicas of tablet 582bff493d324ebc965e61609d5ec08b
I20260929 02:06:48.775450 19726 tablet_service.cc:1514] Processing DeleteTablet for tablet 582bff493d324ebc965e61609d5ec08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:48 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:38910
I20260929 02:06:48.776448 20122 tablet_replica.cc:330] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: stopping tablet replica
I20260929 02:06:48.776935 20122 raft_consensus.cc:2229] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:48.776893 19865 tablet_service.cc:1514] Processing DeleteTablet for tablet 582bff493d324ebc965e61609d5ec08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:48 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:33030
I20260929 02:06:48.777544 20122 raft_consensus.cc:2258] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:48.778017 20004 tablet_service.cc:1514] Processing DeleteTablet for tablet 582bff493d324ebc965e61609d5ec08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:48 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:57998
I20260929 02:06:48.778807 20124 tablet_replica.cc:330] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: stopping tablet replica
I20260929 02:06:48.779281 20124 raft_consensus.cc:2229] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:48.779743 20124 raft_consensus.cc:2258] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:48.780048 19589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:48.780777 19589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:48.781929 20123 tablet_replica.cc:330] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: stopping tablet replica
I20260929 02:06:48.782330 20123 raft_consensus.cc:2229] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:48.783026 20123 raft_consensus.cc:2258] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:48.784889 20123 ts_tablet_manager.cc:1905] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:48.788223 19726 tablet_service.cc:1467] Processing CreateTablet for tablet 23016fde5f334672a9736f0a5acdf22c (DEFAULT_TABLE table=test-table [id=a8a0c9e631124dee8f1dd9deb069d4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:48.788712 19726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23016fde5f334672a9736f0a5acdf22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:48.791491 20123 ts_tablet_manager.cc:1918] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:48.791680 20123 log.cc:1196] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/582bff493d324ebc965e61609d5ec08b
I20260929 02:06:48.792047 20004 tablet_service.cc:1467] Processing CreateTablet for tablet 23016fde5f334672a9736f0a5acdf22c (DEFAULT_TABLE table=test-table [id=a8a0c9e631124dee8f1dd9deb069d4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:48.792245 20123 ts_tablet_manager.cc:1939] T 582bff493d324ebc965e61609d5ec08b P 52c0b5d64a2d46ffba760531e8ce4350: Deleting consensus metadata
I20260929 02:06:48.792480 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23016fde5f334672a9736f0a5acdf22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:48.793655 20124 ts_tablet_manager.cc:1905] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:48.794437 20122 ts_tablet_manager.cc:1905] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:48.799000 20122 ts_tablet_manager.cc:1918] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:48.798977 20099 tablet_bootstrap.cc:492] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: Bootstrap starting.
I20260929 02:06:48.799145 20122 log.cc:1196] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/582bff493d324ebc965e61609d5ec08b
I20260929 02:06:48.799535 19575 catalog_manager.cc:4841] TS 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194:33491): tablet 582bff493d324ebc965e61609d5ec08b (table test-table [id=d0fbe03ac325475b973ff024c2aee609]) successfully deleted
I20260929 02:06:48.799677 20122 ts_tablet_manager.cc:1939] T 582bff493d324ebc965e61609d5ec08b P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting consensus metadata
I20260929 02:06:48.800125 19865 tablet_service.cc:1467] Processing CreateTablet for tablet 23016fde5f334672a9736f0a5acdf22c (DEFAULT_TABLE table=test-table [id=a8a0c9e631124dee8f1dd9deb069d4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:48.800560 19865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23016fde5f334672a9736f0a5acdf22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:48.800730 20099 tablet_bootstrap.cc:654] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:48.801471 19574 catalog_manager.cc:4841] TS 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259): tablet 582bff493d324ebc965e61609d5ec08b (table test-table [id=d0fbe03ac325475b973ff024c2aee609]) successfully deleted
I20260929 02:06:48.808224 20099 tablet_bootstrap.cc:492] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: No bootstrap required, opened a new log
I20260929 02:06:48.808516 20099 ts_tablet_manager.cc:1397] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260929 02:06:48.809451 20099 raft_consensus.cc:348] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.809878 20099 raft_consensus.cc:374] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:48.810040 20099 raft_consensus.cc:729] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b57f799c569438a865f3a0f8af4c480, State: Initialized, Role: FOLLOWER
I20260929 02:06:48.810384 20099 consensus_queue.cc:260] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.811082 20124 ts_tablet_manager.cc:1918] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:48.811249 20124 log.cc:1196] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/582bff493d324ebc965e61609d5ec08b
I20260929 02:06:48.811820 20124 ts_tablet_manager.cc:1939] T 582bff493d324ebc965e61609d5ec08b P 4b57f799c569438a865f3a0f8af4c480: Deleting consensus metadata
I20260929 02:06:48.812119 20099 ts_tablet_manager.cc:1428] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:06:48.813519 19576 catalog_manager.cc:4841] TS 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059): tablet 582bff493d324ebc965e61609d5ec08b (table test-table [id=d0fbe03ac325475b973ff024c2aee609]) successfully deleted
I20260929 02:06:48.814000 20100 tablet_bootstrap.cc:492] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: Bootstrap starting.
I20260929 02:06:48.815444 20098 tablet_bootstrap.cc:492] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: Bootstrap starting.
I20260929 02:06:48.815697 20100 tablet_bootstrap.cc:654] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:48.817085 20098 tablet_bootstrap.cc:654] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:48.819461 20100 tablet_bootstrap.cc:492] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: No bootstrap required, opened a new log
I20260929 02:06:48.819639 20100 ts_tablet_manager.cc:1397] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260929 02:06:48.819845 20098 tablet_bootstrap.cc:492] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: No bootstrap required, opened a new log
I20260929 02:06:48.820000 20098 ts_tablet_manager.cc:1397] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:06:48.820360 20100 raft_consensus.cc:348] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.820559 20100 raft_consensus.cc:374] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:48.820662 20100 raft_consensus.cc:729] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a7fd9b71b74a05b45760ca43b7b1cb, State: Initialized, Role: FOLLOWER
I20260929 02:06:48.820770 20098 raft_consensus.cc:348] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.820999 20098 raft_consensus.cc:374] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:48.820947 20100 consensus_queue.cc:260] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.821105 20098 raft_consensus.cc:729] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c0b5d64a2d46ffba760531e8ce4350, State: Initialized, Role: FOLLOWER
I20260929 02:06:48.821555 20100 ts_tablet_manager.cc:1428] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:06:48.822084 20098 consensus_queue.cc:260] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:48.826066 20098 ts_tablet_manager.cc:1428] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260929 02:06:49.088306 20120 raft_consensus.cc:482] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:49.088611 20120 raft_consensus.cc:504] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.089640 20120 leader_election.cc:288] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:49.090226 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23016fde5f334672a9736f0a5acdf22c" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480" is_pre_election: true
I20260929 02:06:49.090205 19745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23016fde5f334672a9736f0a5acdf22c" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" is_pre_election: true
I20260929 02:06:49.090504 20029 raft_consensus.cc:2454] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 0.
I20260929 02:06:49.090518 19745 raft_consensus.cc:2454] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 0.
I20260929 02:06:49.091024 19819 leader_election.cc:302] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b57f799c569438a865f3a0f8af4c480, 52c0b5d64a2d46ffba760531e8ce4350; no voters:
I20260929 02:06:49.091285 20120 raft_consensus.cc:2781] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:49.091416 20120 raft_consensus.cc:482] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:49.091555 20120 raft_consensus.cc:3037] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.094218 20120 raft_consensus.cc:504] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.094918 20120 leader_election.cc:288] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 election: Requested vote from peers 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:49.095494 19745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23016fde5f334672a9736f0a5acdf22c" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb"
I20260929 02:06:49.095533 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23016fde5f334672a9736f0a5acdf22c" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480"
I20260929 02:06:49.095784 20029 raft_consensus.cc:3037] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.095778 19745 raft_consensus.cc:3037] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.098958 19745 raft_consensus.cc:2454] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 1.
I20260929 02:06:49.098958 20029 raft_consensus.cc:2454] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 1.
I20260929 02:06:49.099488 19817 leader_election.cc:302] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c0b5d64a2d46ffba760531e8ce4350, 82a7fd9b71b74a05b45760ca43b7b1cb; no voters:
I20260929 02:06:49.099845 20120 raft_consensus.cc:2781] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:49.100026 20120 raft_consensus.cc:686] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 LEADER]: Becoming Leader. State: Replica: 52c0b5d64a2d46ffba760531e8ce4350, State: Running, Role: LEADER
I20260929 02:06:49.100329 20120 consensus_queue.cc:237] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.103669 19588 catalog_manager.cc:5495] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194). New cstate: current_term: 1 leader_uuid: "52c0b5d64a2d46ffba760531e8ce4350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:06:49 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647606, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.254@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:49.224822 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.205450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:32873 (local address 127.0.0.1:42008)
0929 02:06:49.206193 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:49.206210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:49.206330 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:49.206663 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:49.206666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:49.207405 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:49.207598 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.207607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.208775 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.208779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.209880 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.209891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.210729 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.210735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.210890 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.216619 (+ 5729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:49.216641 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:49.219093 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:49.222056 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:49.222067 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:49.222074 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:49.222659 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:49.223044 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:49.223047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:49.223050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:49.223256 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:49.223783 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:49.223790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:49.223940 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 02:06:49.224099 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 02:06:49.224328 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":386,"threads_started":1}
I20260929 02:06:49.224939 20135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.205476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.11.182.254:32873)
0929 02:06:49.206244 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:49.206248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:49.206273 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:06:49.206440 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:49.206444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:49.206506 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:49.206605 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:49.207769 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.208560 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.210014 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.210464 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.210617 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.219340 (+ 8723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:49.219345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:49.219349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:49.219382 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:49.221899 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:49.222789 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:49.222794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:49.222796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:49.222934 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:49.223374 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:49.223377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:49.223379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:49.223626 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:49.223742 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:49.224214 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 02:06:49.224483 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":649,"thread_start_us":452,"threads_started":1}
I20260929 02:06:49.230979 19588 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:42008:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:06:49.232074 19588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:49.239362 19726 tablet_service.cc:1467] Processing CreateTablet for tablet b3f5fa4a41274b7f95bb0778bb1e40a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d94fc52cdb4c4a55b54d8db034fb0e5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:06:49.239357 19865 tablet_service.cc:1467] Processing CreateTablet for tablet b3f5fa4a41274b7f95bb0778bb1e40a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d94fc52cdb4c4a55b54d8db034fb0e5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:06:49.239921 19865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f5fa4a41274b7f95bb0778bb1e40a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:49.239954 19726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f5fa4a41274b7f95bb0778bb1e40a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:49.243431 20004 tablet_service.cc:1467] Processing CreateTablet for tablet b3f5fa4a41274b7f95bb0778bb1e40a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d94fc52cdb4c4a55b54d8db034fb0e5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:06:49.244004 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f5fa4a41274b7f95bb0778bb1e40a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:49.247248 20100 tablet_bootstrap.cc:492] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: Bootstrap starting.
I20260929 02:06:49.247251 20098 tablet_bootstrap.cc:492] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: Bootstrap starting.
I20260929 02:06:49.248611 20100 tablet_bootstrap.cc:654] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:49.248921 20098 tablet_bootstrap.cc:654] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:49.251108 20098 tablet_bootstrap.cc:492] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: No bootstrap required, opened a new log
I20260929 02:06:49.251183 20099 tablet_bootstrap.cc:492] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: Bootstrap starting.
I20260929 02:06:49.251293 20098 ts_tablet_manager.cc:1397] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:49.251606 20100 tablet_bootstrap.cc:492] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: No bootstrap required, opened a new log
I20260929 02:06:49.251780 20100 ts_tablet_manager.cc:1397] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:06:49.252065 20098 raft_consensus.cc:348] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.252316 20098 raft_consensus.cc:374] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:49.252398 20098 raft_consensus.cc:729] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c0b5d64a2d46ffba760531e8ce4350, State: Initialized, Role: FOLLOWER
I20260929 02:06:49.252496 20100 raft_consensus.cc:348] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.252660 20098 consensus_queue.cc:260] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.252775 20099 tablet_bootstrap.cc:654] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:49.253190 20098 ts_tablet_manager.cc:1428] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:06:49.253509 20120 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.253770 20120 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:49.253850 20100 raft_consensus.cc:374] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:49.254101 20100 raft_consensus.cc:729] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a7fd9b71b74a05b45760ca43b7b1cb, State: Initialized, Role: FOLLOWER
I20260929 02:06:49.254422 20100 consensus_queue.cc:260] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.254789 20106 raft_consensus.cc:482] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:49.254987 20099 tablet_bootstrap.cc:492] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: No bootstrap required, opened a new log
I20260929 02:06:49.255132 20099 ts_tablet_manager.cc:1397] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 02:06:49.254987 20106 raft_consensus.cc:504] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.255817 20099 raft_consensus.cc:348] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.256049 20099 raft_consensus.cc:374] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:49.256117 20099 raft_consensus.cc:729] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b57f799c569438a865f3a0f8af4c480, State: Initialized, Role: FOLLOWER
I20260929 02:06:49.256561 20099 consensus_queue.cc:260] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.256865 20106 leader_election.cc:288] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194:33491), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:49.257103 20099 ts_tablet_manager.cc:1428] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:06:49.257177 20106 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.257450 20106 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:49.259029 20105 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.259369 20105 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:49.271701 20100 ts_tablet_manager.cc:1428] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent starting tablet: real 0.020s user 0.004s sys 0.012s
I20260929 02:06:49.277077 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.259769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:49083 (local address 127.11.182.195:44059)
0929 02:06:49.263244 (+ 3475us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:49.263252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:49.270479 (+ 7227us) server_negotiation.cc:407] Connection header received
0929 02:06:49.270762 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:49.270767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:49.270865 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:49.270977 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:49.271724 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.272699 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.275043 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.276087 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.276262 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.276386 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:49.276659 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 02:06:49.276797 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3351,"thread_start_us":3230,"threads_started":1}
I20260929 02:06:49.277081 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.258591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:44059 (local address 127.11.182.193:49083)
0929 02:06:49.270354 (+ 11763us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:49.270373 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:49.270555 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:49.271191 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:49.271196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:49.271222 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:49.271566 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.271575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.272867 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.272873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.274800 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.274813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.276323 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.276331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.276479 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.276504 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:06:49.276618 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 02:06:49.276800 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11636,"thread_start_us":12064,"threads_started":1}
I20260929 02:06:49.277885 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f5fa4a41274b7f95bb0778bb1e40a5" candidate_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480" is_pre_election: true
I20260929 02:06:49.278156 20029 raft_consensus.cc:2454] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82a7fd9b71b74a05b45760ca43b7b1cb in term 0.
I20260929 02:06:49.278578 19679 leader_election.cc:302] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b57f799c569438a865f3a0f8af4c480, 82a7fd9b71b74a05b45760ca43b7b1cb; no voters:
I20260929 02:06:49.279040 20139 raft_consensus.cc:2781] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:49.279186 20139 raft_consensus.cc:482] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:49.279347 20139 raft_consensus.cc:3037] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.281641 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.259882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:57565 (local address 127.11.182.194:33491)
0929 02:06:49.275263 (+ 15381us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:49.275269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:49.275287 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:06:49.275350 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:49.275354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:49.275425 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:49.275533 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:06:49.276065 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.277141 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.279463 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.280670 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.280842 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.280923 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:49.281202 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 02:06:49.281336 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15250,"thread_start_us":15139,"threads_started":1}
I20260929 02:06:49.281684 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.257953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:33491 (local address 127.11.182.193:57565)
0929 02:06:49.258570 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:49.258597 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:49.258766 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:49.275598 (+ 16832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:49.275603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:49.275628 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:06:49.275903 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.275912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.277305 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.277310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.279216 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.279232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.280886 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.280891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.281005 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.281028 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:06:49.281171 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 02:06:49.281314 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":2929,"threads_started":1}
I20260929 02:06:49.282599 19890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f5fa4a41274b7f95bb0778bb1e40a5" candidate_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c0b5d64a2d46ffba760531e8ce4350" is_pre_election: true
I20260929 02:06:49.282585 20139 raft_consensus.cc:504] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.283025 19890 raft_consensus.cc:2454] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82a7fd9b71b74a05b45760ca43b7b1cb in term 0.
I20260929 02:06:49.283599 20139 leader_election.cc:288] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [CANDIDATE]: Term 1 election: Requested vote from peers 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194:33491), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:49.283887 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f5fa4a41274b7f95bb0778bb1e40a5" candidate_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480"
I20260929 02:06:49.283911 19890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f5fa4a41274b7f95bb0778bb1e40a5" candidate_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c0b5d64a2d46ffba760531e8ce4350"
I20260929 02:06:49.284098 19890 raft_consensus.cc:3037] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.284124 20029 raft_consensus.cc:3037] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.286087 20029 raft_consensus.cc:2454] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82a7fd9b71b74a05b45760ca43b7b1cb in term 1.
I20260929 02:06:49.286120 19890 raft_consensus.cc:2454] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82a7fd9b71b74a05b45760ca43b7b1cb in term 1.
I20260929 02:06:49.286547 19679 leader_election.cc:302] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b57f799c569438a865f3a0f8af4c480, 82a7fd9b71b74a05b45760ca43b7b1cb; no voters:
I20260929 02:06:49.286934 20139 raft_consensus.cc:2781] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:49.287171 20139 raft_consensus.cc:686] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 LEADER]: Becoming Leader. State: Replica: 82a7fd9b71b74a05b45760ca43b7b1cb, State: Running, Role: LEADER
I20260929 02:06:49.287544 20139 consensus_queue.cc:237] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.289530 20106 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: TxnStatusTablet state changed. Reason: New leader 82a7fd9b71b74a05b45760ca43b7b1cb. Latest consensus state: current_term: 1 leader_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.289921 20106 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:06:49.290611 20145 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:06:49.290889 20145 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:06:49.291759 19588 catalog_manager.cc:5495] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193). New cstate: current_term: 1 leader_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } health_report { overall_health: UNKNOWN } } }
I20260929 02:06:49.325093 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.315046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:46259 (local address 127.0.0.1:38914)
0929 02:06:49.315280 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:49.315300 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:49.315437 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:49.319225 (+ 3788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:49.319231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:49.319260 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:06:49.319464 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.319475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.321043 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.321048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.322066 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.322079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.323112 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.323117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.323274 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.323389 (+ 115us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:06:49.324629 (+ 1240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:06:49.324651 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:06:49.324729 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:06:49.324868 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 02:06:49.325258 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.315121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.11.182.193:46259)
0929 02:06:49.318659 (+ 3538us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:49.318666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:49.318695 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:06:49.318805 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:49.318812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:49.318952 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:49.319134 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:49.319671 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.320885 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.322231 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.322957 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.323092 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.323664 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:49.324471 (+ 807us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:49.324625 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:49.324782 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 02:06:49.324881 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3382,"thread_start_us":3244,"threads_started":1}
I20260929 02:06:49.330003 20029 raft_consensus.cc:1261] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Refusing update from remote peer 82a7fd9b71b74a05b45760ca43b7b1cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:49.330098 19890 raft_consensus.cc:1261] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Refusing update from remote peer 82a7fd9b71b74a05b45760ca43b7b1cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:49.330772 20106 consensus_queue.cc:1035] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:49.331166 20139 consensus_queue.cc:1035] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:49.335085 20120 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: TxnStatusTablet state changed. Reason: New leader 82a7fd9b71b74a05b45760ca43b7b1cb. Latest consensus state: current_term: 1 leader_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.335462 20120 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:49.339381 20149 mvcc.cc:205] Tried to move back new op lower bound from 7334492607808016384 to 7334492607645593600. Current Snapshot: MvccSnapshot[applied={T|T < 7334492607808016384}]
I20260929 02:06:49.343706 20120 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.344008 20120 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 52c0b5d64a2d46ffba760531e8ce4350: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:49.349130 20105 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: TxnStatusTablet state changed. Reason: New leader 82a7fd9b71b74a05b45760ca43b7b1cb. Latest consensus state: current_term: 1 leader_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.349413 20105 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:49.355173 20106 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.355618 20106 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:06:49.358287 20106 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.358637 20106 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 82a7fd9b71b74a05b45760ca43b7b1cb: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:06:49.361845 20105 tablet_replica.cc:439] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } } }
I20260929 02:06:49.362133 20105 tablet_replica.cc:442] T b3f5fa4a41274b7f95bb0778bb1e40a5 P 4b57f799c569438a865f3a0f8af4c480: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:06:49.382594 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.376155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:45001 (local address 127.11.182.193:46259)
0929 02:06:49.376366 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:49.376372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:49.376799 (+ 427us) server_negotiation.cc:407] Connection header received
0929 02:06:49.377090 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:49.377095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:49.377162 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:49.377252 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:49.377715 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.378538 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.380567 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.380997 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.381115 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.381801 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:49.381915 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:49.381995 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:49.382217 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 02:06:49.382284 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:06:49.382776 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.375978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:46259 (local address 127.11.182.194:45001)
0929 02:06:49.376661 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:49.376682 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:49.376853 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:49.377290 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:49.377295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:49.377308 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:06:49.377565 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.377576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.378691 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.378697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.380401 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.380418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.381171 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.381177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.381376 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.381458 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:06:49.382080 (+ 622us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:06:49.382093 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 02:06:49.382189 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:06:49.382358 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":286,"threads_started":1}
I20260929 02:06:49.392741 19745 raft_consensus.cc:1261] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Refusing update from remote peer 52c0b5d64a2d46ffba760531e8ce4350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:49.392724 20029 raft_consensus.cc:1261] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Refusing update from remote peer 52c0b5d64a2d46ffba760531e8ce4350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:49.393302 20109 consensus_queue.cc:1035] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:49.393599 20120 consensus_queue.cc:1035] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:49.439718 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.432907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:34133 (local address 127.11.182.194:33491)
0929 02:06:49.433078 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:49.433084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:49.433850 (+ 766us) server_negotiation.cc:407] Connection header received
0929 02:06:49.434086 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:49.434091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:49.434163 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:49.434259 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:49.434769 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.435709 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.437894 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.438336 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.438441 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.439079 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:49.439200 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:49.439284 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:49.439453 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 02:06:49.439519 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:06:49.440099 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.432955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:33491 (local address 127.11.182.193:34133)
0929 02:06:49.433740 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:49.433755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:49.433933 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:49.434274 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:49.434279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:49.434290 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:06:49.434622 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.434631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.435847 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.435853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.437730 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:49.437745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.438533 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.438541 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:49.438727 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.438808 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:06:49.439308 (+ 500us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:06:49.439332 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:06:49.439428 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:06:49.439580 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":478,"threads_started":1}
I20260929 02:06:49.476365 19588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:49.476600 19588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:49.479707 19588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6: Sending DeleteTablet for 3 replicas of tablet 23016fde5f334672a9736f0a5acdf22c
I20260929 02:06:49.480510 19865 tablet_service.cc:1514] Processing DeleteTablet for tablet 23016fde5f334672a9736f0a5acdf22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:49 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:33030
I20260929 02:06:49.480532 19726 tablet_service.cc:1514] Processing DeleteTablet for tablet 23016fde5f334672a9736f0a5acdf22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:49 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:38910
I20260929 02:06:49.481772 20004 tablet_service.cc:1514] Processing DeleteTablet for tablet 23016fde5f334672a9736f0a5acdf22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:49 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:57998
I20260929 02:06:49.490860 20170 tablet_replica.cc:330] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: stopping tablet replica
I20260929 02:06:49.491101 20168 tablet_replica.cc:330] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: stopping tablet replica
I20260929 02:06:49.491087 20170 raft_consensus.cc:2229] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:49.491360 20170 raft_consensus.cc:2258] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:49.491384 20168 raft_consensus.cc:2229] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:49.492317 20168 raft_consensus.cc:2258] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:49.493458 20168 ts_tablet_manager.cc:1905] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:49.493463 20170 ts_tablet_manager.cc:1905] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:49.500694 20168 ts_tablet_manager.cc:1918] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:49.500963 20168 log.cc:1196] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/23016fde5f334672a9736f0a5acdf22c
I20260929 02:06:49.501498 20168 ts_tablet_manager.cc:1939] T 23016fde5f334672a9736f0a5acdf22c P 52c0b5d64a2d46ffba760531e8ce4350: Deleting consensus metadata
I20260929 02:06:49.502738 19575 catalog_manager.cc:4841] TS 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194:33491): tablet 23016fde5f334672a9736f0a5acdf22c (table test-table [id=a8a0c9e631124dee8f1dd9deb069d4fc]) successfully deleted
I20260929 02:06:49.503599 20169 tablet_replica.cc:330] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: stopping tablet replica
I20260929 02:06:49.503841 20169 raft_consensus.cc:2229] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:49.504268 20169 raft_consensus.cc:2258] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:49.505388 20169 ts_tablet_manager.cc:1905] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:49.506937 20170 ts_tablet_manager.cc:1918] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:49.507085 20170 log.cc:1196] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/23016fde5f334672a9736f0a5acdf22c
I20260929 02:06:49.507553 20170 ts_tablet_manager.cc:1939] T 23016fde5f334672a9736f0a5acdf22c P 4b57f799c569438a865f3a0f8af4c480: Deleting consensus metadata
I20260929 02:06:49.508801 19576 catalog_manager.cc:4841] TS 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059): tablet 23016fde5f334672a9736f0a5acdf22c (table test-table [id=a8a0c9e631124dee8f1dd9deb069d4fc]) successfully deleted
I20260929 02:06:49.513361 20169 ts_tablet_manager.cc:1918] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:49.513594 20169 log.cc:1196] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/23016fde5f334672a9736f0a5acdf22c
I20260929 02:06:49.514096 20169 ts_tablet_manager.cc:1939] T 23016fde5f334672a9736f0a5acdf22c P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting consensus metadata
I20260929 02:06:49.515275 19574 catalog_manager.cc:4841] TS 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259): tablet 23016fde5f334672a9736f0a5acdf22c (table test-table [id=a8a0c9e631124dee8f1dd9deb069d4fc]) successfully deleted
Sep 29 02:06:49 dist-test-slave-q06p krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:06:49 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:49 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:06:49.543968 20135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:49.532672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.11.182.254:32873)
0929 02:06:49.532899 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:49.532906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:49.533242 (+ 336us) server_negotiation.cc:407] Connection header received
0929 02:06:49.533368 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:49.533375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:49.533461 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:49.533576 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:49.534891 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.536037 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.537333 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.537976 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.538115 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.543700 (+ 5585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42022: BlockingRecv error: recv got EOF from 127.0.0.1:42022 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 02:06:49 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM
I20260929 02:06:49.562484 20135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.546675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.11.182.254:32873)
0929 02:06:49.546879 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:49.546885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:49.547208 (+ 323us) server_negotiation.cc:407] Connection header received
0929 02:06:49.547337 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:49.547343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:49.547419 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:49.547523 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:49.548596 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.549731 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.551099 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.551755 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.551880 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.555580 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:49.555586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:49.555591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:49.555631 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:49.558962 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:49.560018 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:49.560026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:49.560030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:49.560203 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:49.560730 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:49.560737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:49.560741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:49.561281 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:49.561545 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:49.561963 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 02:06:49.562164 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:06:49.566526 19588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:06:49.567269 19588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:49.573917 19726 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4cb4ae15de4070a3a1133b496aa43f (DEFAULT_TABLE table=test-table [id=de16857abe574d1f9be23f2ace1c3ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:49.574205 19865 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4cb4ae15de4070a3a1133b496aa43f (DEFAULT_TABLE table=test-table [id=de16857abe574d1f9be23f2ace1c3ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:49.574383 19726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4cb4ae15de4070a3a1133b496aa43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:49.574601 19865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4cb4ae15de4070a3a1133b496aa43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:49.575241 20004 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4cb4ae15de4070a3a1133b496aa43f (DEFAULT_TABLE table=test-table [id=de16857abe574d1f9be23f2ace1c3ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:49.575608 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4cb4ae15de4070a3a1133b496aa43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:49.581765 20100 tablet_bootstrap.cc:492] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: Bootstrap starting.
I20260929 02:06:49.582485 20098 tablet_bootstrap.cc:492] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: Bootstrap starting.
I20260929 02:06:49.583609 20100 tablet_bootstrap.cc:654] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:49.583796 20098 tablet_bootstrap.cc:654] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:49.585844 20099 tablet_bootstrap.cc:492] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: Bootstrap starting.
I20260929 02:06:49.586203 20098 tablet_bootstrap.cc:492] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: No bootstrap required, opened a new log
I20260929 02:06:49.586339 20098 ts_tablet_manager.cc:1397] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:06:49.587459 20099 tablet_bootstrap.cc:654] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:49.587644 20098 raft_consensus.cc:348] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.587932 20098 raft_consensus.cc:374] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:49.588039 20098 raft_consensus.cc:729] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c0b5d64a2d46ffba760531e8ce4350, State: Initialized, Role: FOLLOWER
I20260929 02:06:49.588341 20098 consensus_queue.cc:260] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.592576 20098 ts_tablet_manager.cc:1428] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 02:06:49.594776 20100 tablet_bootstrap.cc:492] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: No bootstrap required, opened a new log
I20260929 02:06:49.595057 20100 ts_tablet_manager.cc:1397] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260929 02:06:49.596035 20100 raft_consensus.cc:348] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.596343 20100 raft_consensus.cc:374] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:49.596455 20100 raft_consensus.cc:729] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a7fd9b71b74a05b45760ca43b7b1cb, State: Initialized, Role: FOLLOWER
I20260929 02:06:49.596747 20100 consensus_queue.cc:260] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.597363 20100 ts_tablet_manager.cc:1428] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:06:49.601022 20099 tablet_bootstrap.cc:492] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: No bootstrap required, opened a new log
I20260929 02:06:49.601200 20099 ts_tablet_manager.cc:1397] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260929 02:06:49.602095 20099 raft_consensus.cc:348] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.602357 20099 raft_consensus.cc:374] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:49.602451 20099 raft_consensus.cc:729] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b57f799c569438a865f3a0f8af4c480, State: Initialized, Role: FOLLOWER
I20260929 02:06:49.602710 20099 consensus_queue.cc:260] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.606149 20099 ts_tablet_manager.cc:1428] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:06:49.765015 20109 raft_consensus.cc:482] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:49.765251 20109 raft_consensus.cc:504] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.766042 20109 leader_election.cc:288] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:49.766599 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4cb4ae15de4070a3a1133b496aa43f" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" is_pre_election: true
I20260929 02:06:49.766710 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4cb4ae15de4070a3a1133b496aa43f" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480" is_pre_election: true
I20260929 02:06:49.766955 19746 raft_consensus.cc:2454] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 0.
I20260929 02:06:49.766983 20028 raft_consensus.cc:2454] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 0.
I20260929 02:06:49.767508 19819 leader_election.cc:302] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b57f799c569438a865f3a0f8af4c480, 52c0b5d64a2d46ffba760531e8ce4350; no voters:
I20260929 02:06:49.767839 20109 raft_consensus.cc:2781] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:49.767987 20109 raft_consensus.cc:482] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:49.768086 20109 raft_consensus.cc:3037] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.770913 20109 raft_consensus.cc:504] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.771587 20109 leader_election.cc:288] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 election: Requested vote from peers 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:49.772146 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4cb4ae15de4070a3a1133b496aa43f" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb"
I20260929 02:06:49.772379 19746 raft_consensus.cc:3037] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.772313 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4cb4ae15de4070a3a1133b496aa43f" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480"
I20260929 02:06:49.772509 20028 raft_consensus.cc:3037] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.776047 20028 raft_consensus.cc:2454] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 1.
I20260929 02:06:49.776031 19746 raft_consensus.cc:2454] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 1.
I20260929 02:06:49.777088 19817 leader_election.cc:302] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b57f799c569438a865f3a0f8af4c480, 52c0b5d64a2d46ffba760531e8ce4350, 82a7fd9b71b74a05b45760ca43b7b1cb; no voters:
I20260929 02:06:49.777827 20109 raft_consensus.cc:2781] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:49.778096 20109 raft_consensus.cc:686] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 LEADER]: Becoming Leader. State: Replica: 52c0b5d64a2d46ffba760531e8ce4350, State: Running, Role: LEADER
I20260929 02:06:49.778470 20109 consensus_queue.cc:237] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.781744 19589 catalog_manager.cc:5495] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194). New cstate: current_term: 1 leader_uuid: "52c0b5d64a2d46ffba760531e8ce4350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } health_report { overall_health: UNKNOWN } } }
I20260929 02:06:49.798647 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:49.792775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.11.182.194:33491)
0929 02:06:49.793055 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:49.793061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:49.793084 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:06:49.793217 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:49.793223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:49.793304 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:49.793398 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:06:49.793869 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.794692 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.796631 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:49.797098 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:49.797222 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:49.797799 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:06:49.797913 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:06:49.797991 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:49.798184 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 02:06:49.798266 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260929 02:06:49.801543 20028 raft_consensus.cc:1261] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Refusing update from remote peer 52c0b5d64a2d46ffba760531e8ce4350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:49.801580 19746 raft_consensus.cc:1261] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Refusing update from remote peer 52c0b5d64a2d46ffba760531e8ce4350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:49.802194 20167 consensus_queue.cc:1035] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:49.802546 20109 consensus_queue.cc:1035] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:49.816386 19588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:49.817025 19588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:49.820580 19588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6: Sending DeleteTablet for 3 replicas of tablet 7d4cb4ae15de4070a3a1133b496aa43f
I20260929 02:06:49.821569 19726 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4cb4ae15de4070a3a1133b496aa43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:49 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:38910
I20260929 02:06:49.821689 20004 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4cb4ae15de4070a3a1133b496aa43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:49 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:57998
I20260929 02:06:49.821739 19865 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4cb4ae15de4070a3a1133b496aa43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:49 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:33030
I20260929 02:06:49.821870 20169 tablet_replica.cc:330] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: stopping tablet replica
I20260929 02:06:49.821970 20170 tablet_replica.cc:330] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: stopping tablet replica
I20260929 02:06:49.822016 20168 tablet_replica.cc:330] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: stopping tablet replica
I20260929 02:06:49.822080 20169 raft_consensus.cc:2229] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:49.822168 20170 raft_consensus.cc:2229] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:49.822208 20168 raft_consensus.cc:2229] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:49.822319 20169 raft_consensus.cc:2258] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:49.822402 20170 raft_consensus.cc:2258] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:49.822521 20168 raft_consensus.cc:2258] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:49.823525 20169 ts_tablet_manager.cc:1905] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:49.823740 20170 ts_tablet_manager.cc:1905] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:49.823551 19588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 02:06:49.824380 20168 ts_tablet_manager.cc:1905] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:06:49.824558 19588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:06:49.830478 20170 ts_tablet_manager.cc:1918] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:49.830644 20170 log.cc:1196] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/7d4cb4ae15de4070a3a1133b496aa43f
I20260929 02:06:49.831110 20170 ts_tablet_manager.cc:1939] T 7d4cb4ae15de4070a3a1133b496aa43f P 4b57f799c569438a865f3a0f8af4c480: Deleting consensus metadata
I20260929 02:06:49.831209 20168 ts_tablet_manager.cc:1918] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:49.831357 20168 log.cc:1196] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/7d4cb4ae15de4070a3a1133b496aa43f
I20260929 02:06:49.831749 20168 ts_tablet_manager.cc:1939] T 7d4cb4ae15de4070a3a1133b496aa43f P 52c0b5d64a2d46ffba760531e8ce4350: Deleting consensus metadata
I20260929 02:06:49.832389 19576 catalog_manager.cc:4841] TS 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059): tablet 7d4cb4ae15de4070a3a1133b496aa43f (table test-table [id=de16857abe574d1f9be23f2ace1c3ffd]) successfully deleted
I20260929 02:06:49.833010 19575 catalog_manager.cc:4841] TS 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194:33491): tablet 7d4cb4ae15de4070a3a1133b496aa43f (table test-table [id=de16857abe574d1f9be23f2ace1c3ffd]) successfully deleted
I20260929 02:06:49.833235 19726 tablet_service.cc:1467] Processing CreateTablet for tablet 6bcf3325ced14002a0f9cd39c118d979 (DEFAULT_TABLE table=test-table [id=c7d177fdfc254fd7b4c30b951dadeb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:49.833691 19865 tablet_service.cc:1467] Processing CreateTablet for tablet 6bcf3325ced14002a0f9cd39c118d979 (DEFAULT_TABLE table=test-table [id=c7d177fdfc254fd7b4c30b951dadeb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:49.833909 19726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcf3325ced14002a0f9cd39c118d979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:49.834084 19865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcf3325ced14002a0f9cd39c118d979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:49.834362 20004 tablet_service.cc:1467] Processing CreateTablet for tablet 6bcf3325ced14002a0f9cd39c118d979 (DEFAULT_TABLE table=test-table [id=c7d177fdfc254fd7b4c30b951dadeb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:06:49.834784 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcf3325ced14002a0f9cd39c118d979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:49.836107 20169 ts_tablet_manager.cc:1918] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:06:49.836629 20169 log.cc:1196] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/7d4cb4ae15de4070a3a1133b496aa43f
I20260929 02:06:49.837149 20169 ts_tablet_manager.cc:1939] T 7d4cb4ae15de4070a3a1133b496aa43f P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting consensus metadata
I20260929 02:06:49.838469 19574 catalog_manager.cc:4841] TS 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259): tablet 7d4cb4ae15de4070a3a1133b496aa43f (table test-table [id=de16857abe574d1f9be23f2ace1c3ffd]) successfully deleted
I20260929 02:06:49.841233 20100 tablet_bootstrap.cc:492] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: Bootstrap starting.
I20260929 02:06:49.841629 20098 tablet_bootstrap.cc:492] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: Bootstrap starting.
I20260929 02:06:49.842525 20099 tablet_bootstrap.cc:492] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: Bootstrap starting.
I20260929 02:06:49.843197 20100 tablet_bootstrap.cc:654] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:49.843456 20098 tablet_bootstrap.cc:654] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:49.844480 20099 tablet_bootstrap.cc:654] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:49.846175 20100 tablet_bootstrap.cc:492] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: No bootstrap required, opened a new log
I20260929 02:06:49.846395 20100 ts_tablet_manager.cc:1397] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:06:49.847287 20098 tablet_bootstrap.cc:492] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: No bootstrap required, opened a new log
I20260929 02:06:49.847463 20098 ts_tablet_manager.cc:1397] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 02:06:49.847290 20100 raft_consensus.cc:348] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.847575 20100 raft_consensus.cc:374] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:49.847668 20100 raft_consensus.cc:729] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a7fd9b71b74a05b45760ca43b7b1cb, State: Initialized, Role: FOLLOWER
I20260929 02:06:49.847908 20100 consensus_queue.cc:260] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.848301 20098 raft_consensus.cc:348] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.848488 20100 ts_tablet_manager.cc:1428] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:06:49.848546 20098 raft_consensus.cc:374] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:49.848625 20098 raft_consensus.cc:729] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c0b5d64a2d46ffba760531e8ce4350, State: Initialized, Role: FOLLOWER
I20260929 02:06:49.848866 20098 consensus_queue.cc:260] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.849367 20098 ts_tablet_manager.cc:1428] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:06:49.850318 20099 tablet_bootstrap.cc:492] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: No bootstrap required, opened a new log
I20260929 02:06:49.850528 20099 ts_tablet_manager.cc:1397] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 02:06:49.851418 20099 raft_consensus.cc:348] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.851683 20099 raft_consensus.cc:374] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:49.851785 20099 raft_consensus.cc:729] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b57f799c569438a865f3a0f8af4c480, State: Initialized, Role: FOLLOWER
I20260929 02:06:49.852033 20099 consensus_queue.cc:260] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.852648 20099 ts_tablet_manager.cc:1428] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 02:06:49.902545 20120 raft_consensus.cc:482] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:06:49.902797 20120 raft_consensus.cc:504] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.903527 20120 leader_election.cc:288] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:49.904173 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcf3325ced14002a0f9cd39c118d979" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" is_pre_election: true
I20260929 02:06:49.904218 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcf3325ced14002a0f9cd39c118d979" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480" is_pre_election: true
I20260929 02:06:49.904443 19746 raft_consensus.cc:2454] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 0.
I20260929 02:06:49.904493 20028 raft_consensus.cc:2454] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 0.
I20260929 02:06:49.904945 19819 leader_election.cc:302] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b57f799c569438a865f3a0f8af4c480, 52c0b5d64a2d46ffba760531e8ce4350, 82a7fd9b71b74a05b45760ca43b7b1cb; no voters:
I20260929 02:06:49.905303 20120 raft_consensus.cc:2781] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:06:49.905439 20120 raft_consensus.cc:482] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:06:49.905550 20120 raft_consensus.cc:3037] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.908536 20120 raft_consensus.cc:504] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.909490 20120 leader_election.cc:288] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 election: Requested vote from peers 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259), 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059)
I20260929 02:06:49.910197 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcf3325ced14002a0f9cd39c118d979" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb"
I20260929 02:06:49.910192 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcf3325ced14002a0f9cd39c118d979" candidate_uuid: "52c0b5d64a2d46ffba760531e8ce4350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b57f799c569438a865f3a0f8af4c480"
I20260929 02:06:49.910475 19746 raft_consensus.cc:3037] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.910578 20028 raft_consensus.cc:3037] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:49.912699 20028 raft_consensus.cc:2454] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 1.
I20260929 02:06:49.912701 19746 raft_consensus.cc:2454] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c0b5d64a2d46ffba760531e8ce4350 in term 1.
I20260929 02:06:49.913210 19817 leader_election.cc:302] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b57f799c569438a865f3a0f8af4c480, 52c0b5d64a2d46ffba760531e8ce4350, 82a7fd9b71b74a05b45760ca43b7b1cb; no voters:
I20260929 02:06:49.913632 20120 raft_consensus.cc:2781] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:49.913792 20120 raft_consensus.cc:686] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 LEADER]: Becoming Leader. State: Replica: 52c0b5d64a2d46ffba760531e8ce4350, State: Running, Role: LEADER
I20260929 02:06:49.914083 20120 consensus_queue.cc:237] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } }
I20260929 02:06:49.917073 19587 catalog_manager.cc:5495] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194). New cstate: current_term: 1 leader_uuid: "52c0b5d64a2d46ffba760531e8ce4350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c0b5d64a2d46ffba760531e8ce4350" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 } health_report { overall_health: UNKNOWN } } }
I20260929 02:06:49.949494 20028 raft_consensus.cc:1261] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Refusing update from remote peer 52c0b5d64a2d46ffba760531e8ce4350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:49.949520 19746 raft_consensus.cc:1261] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Refusing update from remote peer 52c0b5d64a2d46ffba760531e8ce4350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:06:49.950075 20109 consensus_queue.cc:1035] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b57f799c569438a865f3a0f8af4c480" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:49.950371 20120 consensus_queue.cc:1035] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a7fd9b71b74a05b45760ca43b7b1cb" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:06:50.004103 19587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:50.004419 19587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:06:50.007766 19587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02e7babdf14a487d96d7c6428f6c90a6: Sending DeleteTablet for 3 replicas of tablet 6bcf3325ced14002a0f9cd39c118d979
I20260929 02:06:50.008824 19865 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bcf3325ced14002a0f9cd39c118d979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:50 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:33030
I20260929 02:06:50.008862 19726 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bcf3325ced14002a0f9cd39c118d979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:50 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:38910
I20260929 02:06:50.008847 20004 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bcf3325ced14002a0f9cd39c118d979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:06:50 UTC) from {username='oryx', principal='oryx/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:57998
I20260929 02:06:50.009198 20168 tablet_replica.cc:330] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: stopping tablet replica
I20260929 02:06:50.009222 20170 tablet_replica.cc:330] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: stopping tablet replica
I20260929 02:06:50.009320 20169 tablet_replica.cc:330] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: stopping tablet replica
I20260929 02:06:50.009416 20168 raft_consensus.cc:2229] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:06:50.009424 20170 raft_consensus.cc:2229] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:50.009485 20169 raft_consensus.cc:2229] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:06:50.009678 20170 raft_consensus.cc:2258] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:50.009682 20169 raft_consensus.cc:2258] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:50.009747 20168 raft_consensus.cc:2258] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:06:50.011296 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 19660
I20260929 02:06:50.011286 20168 ts_tablet_manager.cc:1905] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:50.011289 20170 ts_tablet_manager.cc:1905] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:50.013465 20169 ts_tablet_manager.cc:1905] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:06:50.018810 20170 ts_tablet_manager.cc:1918] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:50.019008 20170 log.cc:1196] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/6bcf3325ced14002a0f9cd39c118d979
I20260929 02:06:50.019526 20170 ts_tablet_manager.cc:1939] T 6bcf3325ced14002a0f9cd39c118d979 P 4b57f799c569438a865f3a0f8af4c480: Deleting consensus metadata
I20260929 02:06:50.020792 20169 ts_tablet_manager.cc:1918] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:50.020955 20169 log.cc:1196] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/6bcf3325ced14002a0f9cd39c118d979
I20260929 02:06:50.020919 19576 catalog_manager.cc:4841] TS 4b57f799c569438a865f3a0f8af4c480 (127.11.182.195:44059): tablet 6bcf3325ced14002a0f9cd39c118d979 (table test-table [id=c7d177fdfc254fd7b4c30b951dadeb68]) successfully deleted
I20260929 02:06:50.021561 20169 ts_tablet_manager.cc:1939] T 6bcf3325ced14002a0f9cd39c118d979 P 82a7fd9b71b74a05b45760ca43b7b1cb: Deleting consensus metadata
I20260929 02:06:50.022751 19574 catalog_manager.cc:4841] TS 82a7fd9b71b74a05b45760ca43b7b1cb (127.11.182.193:46259): tablet 6bcf3325ced14002a0f9cd39c118d979 (table test-table [id=c7d177fdfc254fd7b4c30b951dadeb68]) successfully deleted
I20260929 02:06:50.025439 20168 ts_tablet_manager.cc:1918] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:06:50.025619 20168 log.cc:1196] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/6bcf3325ced14002a0f9cd39c118d979
I20260929 02:06:50.026109 20168 ts_tablet_manager.cc:1939] T 6bcf3325ced14002a0f9cd39c118d979 P 52c0b5d64a2d46ffba760531e8ce4350: Deleting consensus metadata
I20260929 02:06:50.027302 19575 catalog_manager.cc:4841] TS 52c0b5d64a2d46ffba760531e8ce4350 (127.11.182.194:33491): tablet 6bcf3325ced14002a0f9cd39c118d979 (table test-table [id=c7d177fdfc254fd7b4c30b951dadeb68]) successfully deleted
Sep 29 02:06:50 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.193@KRBTEST.COM
I20260929 02:06:50.042551 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:50.013260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.11.182.193:46259)
0929 02:06:50.023606 (+ 10346us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:50.023610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:50.023625 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:06:50.023684 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:50.023687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:50.023748 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:50.023853 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:50.025794 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:50.026961 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:50.028702 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:50.029265 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:50.029526 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:50.034725 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:50.034748 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:50.034766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:50.034815 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:50.038506 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:50.039946 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:50.039955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:50.039957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:50.040272 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:50.040794 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:50.040798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:50.040801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:50.041187 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:50.041435 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:50.041863 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 02:06:50.042219 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10241,"thread_start_us":10152,"threads_started":1}
I20260929 02:06:50.043411 19791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:50.171267 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 19660
I20260929 02:06:50.196142 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 19800
Sep 29 02:06:50 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.194@KRBTEST.COM
I20260929 02:06:50.212532 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:50.196896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.11.182.194:33491)
0929 02:06:50.197081 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:50.197085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:50.197103 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:06:50.197184 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:50.197189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:50.197249 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:50.197351 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:50.198561 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:50.199409 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:50.200922 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:50.201396 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:50.201527 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:50.206225 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:50.206240 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:50.206257 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:50.206308 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:50.209282 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:50.210154 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:50.210163 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:50.210165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:50.210315 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:50.210701 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:50.210706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:50.210711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:50.211241 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:50.211581 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:50.211902 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 02:06:50.212157 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:06:50.213317 19930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:50.361799 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 19800
I20260929 02:06:50.388679 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 19939
Sep 29 02:06:50 dist-test-slave-q06p krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.182.195@KRBTEST.COM
I20260929 02:06:50.407102 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:50.389574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.11.182.195:44059)
0929 02:06:50.390141 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:50.390145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:50.390161 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:06:50.390217 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:50.390220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:50.390286 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:50.390376 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:50.391755 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:50.392678 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:50.394072 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:50.394590 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:50.394720 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:50.399974 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:50.399992 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:50.400010 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:50.400061 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:50.403749 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:50.404635 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:50.404644 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:50.404646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:50.404811 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:50.405264 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:50.405271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:50.405274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:50.405720 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:50.406042 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:50.406427 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 02:06:50.406683 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":355,"threads_started":1}
I20260929 02:06:50.408061 20069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:50.509339 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 19939
I20260929 02:06:50.529726 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 19557
I20260929 02:06:50.531054 19629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:50.655130 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 19557
2026-09-29T02:06:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5779 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:50 dist-test-slave-q06p krb5kdc[20209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:50 dist-test-slave-q06p krb5kdc[20209](info): set up 2 sockets
Sep 29 02:06:50 dist-test-slave-q06p krb5kdc[20209](info): commencing operation
krb5kdc: starting...
W20260929 02:06:51.812453 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:51 dist-test-slave-q06p krb5kdc[20209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:06:52.051445 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33663
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:33663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:52.380015 20222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:52.380308 20222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:52.380398 20222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:52.387735 20222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:52.387850 20222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:52.387895 20222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:52.387952 20222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:52.388000 20222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:52.397850 20222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:33663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:52.399097 20222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:52.400354 20222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:52.412523 20227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:52.412559 20228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:52.412503 20230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:52.413714 20222 server_base.cc:1034] running on GCE node
I20260929 02:06:52.414523 20222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:52.416038 20222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:52.417279 20222 hybrid_clock.cc:648] HybridClock initialized: now 1790647612417240 us; error 57 us; skew 500 ppm
I20260929 02:06:52.417624 20222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T02:06:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1824 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:06:52 dist-test-slave-q06p krb5kdc[20236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:06:52 dist-test-slave-q06p krb5kdc[20236](info): set up 2 sockets
Sep 29 02:06:52 dist-test-slave-q06p krb5kdc[20236](info): commencing operation
krb5kdc: starting...
W20260929 02:06:53.628401 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:06:53 dist-test-slave-q06p krb5kdc[20236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:06:53.862730 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33325
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:33325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:54.199035 20249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:54.199424 20249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:54.199601 20249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:54.206869 20249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:54.207043 20249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:54.207154 20249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:54.207259 20249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:54.207350 20249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:54.217325 20249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:33325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:54.218758 20249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:54.220153 20249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:54.232615 20254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:54.232688 20257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:54.232692 20255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:54.233551 20249 server_base.cc:1034] running on GCE node
I20260929 02:06:54.234453 20249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:54.236200 20249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:54.237442 20249 hybrid_clock.cc:648] HybridClock initialized: now 1790647614237400 us; error 42 us; skew 500 ppm
I20260929 02:06:54.237805 20249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:54 dist-test-slave-q06p krb5kdc[20236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647614, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:54.255126 20249 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:06:54.256866 20249 webserver.cc:460] Webserver started at http://127.11.182.254:40741/ using document root <none> and password file <none>
I20260929 02:06:54.257638 20249 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:54.257802 20249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:54.258136 20249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:54.261360 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "d7afda6ccc4b4308bd0075eb1df7609b"
format_stamp: "Formatted at 2026-09-29 02:06:54 on dist-test-slave-q06p"
server_key: "fd6857a7164d5a871ef53eaf7c64f0ce"
server_key_iv: "50c6b0d2ea745e58d8481c8149909d1f"
server_key_version: "encryptionkey@0"
I20260929 02:06:54.262188 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "d7afda6ccc4b4308bd0075eb1df7609b"
format_stamp: "Formatted at 2026-09-29 02:06:54 on dist-test-slave-q06p"
server_key: "fd6857a7164d5a871ef53eaf7c64f0ce"
server_key_iv: "50c6b0d2ea745e58d8481c8149909d1f"
server_key_version: "encryptionkey@0"
I20260929 02:06:54.267794 20249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:06:54.271489 20264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:54.272887 20249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:06:54.273092 20249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "d7afda6ccc4b4308bd0075eb1df7609b"
format_stamp: "Formatted at 2026-09-29 02:06:54 on dist-test-slave-q06p"
server_key: "fd6857a7164d5a871ef53eaf7c64f0ce"
server_key_iv: "50c6b0d2ea745e58d8481c8149909d1f"
server_key_version: "encryptionkey@0"
I20260929 02:06:54.273304 20249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:54.290282 20249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:54.294539 20249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:54.294937 20249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:54.314080 20249 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:33325
I20260929 02:06:54.314101 20315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:33325 every 8 connection(s)
I20260929 02:06:54.315563 20249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:54.319608 20316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:06:54.325801 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 20249
I20260929 02:06:54.326037 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:06:54.326808 11995 external_mini_cluster.cc:1428] Setting key d7427d8d3c6770ad34df1485564edae4
I20260929 02:06:54.331462 20316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: Bootstrap starting.
I20260929 02:06:54.335366 20316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: Neither blocks nor log segments found. Creating new log.
I20260929 02:06:54.336735 20316 log.cc:824] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:54.340195 20316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: No bootstrap required, opened a new log
Sep 29 02:06:54 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:54.348847 20316 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } }
I20260929 02:06:54.349274 20316 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:06:54.349414 20316 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7afda6ccc4b4308bd0075eb1df7609b, State: Initialized, Role: FOLLOWER
I20260929 02:06:54.350147 20316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } }
I20260929 02:06:54.350415 20316 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:54.350575 20316 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:54.350775 20316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:06:54.351217 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:54.329071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.11.182.254:33325)
0929 02:06:54.329903 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:54.329915 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:54.329957 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:06:54.331810 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:54.331836 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:54.332251 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:54.332591 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:54.334409 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:54.335933 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:54.337705 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:54.338534 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:54.338673 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:54.343293 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:54.343300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:54.343321 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:54.343363 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:54.346567 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:54.347425 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:54.347432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:54.347439 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:54.347623 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:54.348046 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:54.348051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:54.348054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:54.348663 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:54.348976 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:54.349411 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 02:06:54.349921 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":406,"threads_started":1}
I20260929 02:06:54.353245 20316 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } }
I20260929 02:06:54.353789 20316 leader_election.cc:302] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7afda6ccc4b4308bd0075eb1df7609b; no voters:
I20260929 02:06:54.354470 20316 leader_election.cc:288] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:06:54.354563 20321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:06:54.355348 20321 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 1 LEADER]: Becoming Leader. State: Replica: d7afda6ccc4b4308bd0075eb1df7609b, State: Running, Role: LEADER
I20260929 02:06:54.355888 20321 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } }
I20260929 02:06:54.356714 20316 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:54.364372 20322 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } } }
I20260929 02:06:54.364768 20322 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:54.365192 20323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7afda6ccc4b4308bd0075eb1df7609b. Latest consensus state: current_term: 1 leader_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } } }
I20260929 02:06:54.365558 20323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:54.373277 20331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:54.378358 20331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:54.387470 20331 catalog_manager.cc:1347] Generated new cluster ID: c05cb5dc8c3e463f8a4c40f9c9753035
I20260929 02:06:54.387601 20331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:54.402293 20331 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:06:54.403942 20331 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:54.412949 20331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: Generated new TSK 0
I20260929 02:06:54.413884 20331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:06:54.476740 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33325
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:54.850011 20343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:54.850319 20343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:54.850477 20343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:54.858214 20343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:54.858340 20343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:54.858510 20343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:54.868881 20343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:54.870209 20343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:54.871670 20343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:54.886325 20348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:54.886391 20349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:54.886365 20351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:54.887183 20343 server_base.cc:1034] running on GCE node
I20260929 02:06:54.887951 20343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:54.889122 20343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:54.890372 20343 hybrid_clock.cc:648] HybridClock initialized: now 1790647614890304 us; error 68 us; skew 500 ppm
I20260929 02:06:54.890722 20343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:54 dist-test-slave-q06p krb5kdc[20236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647614, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:54.894052 20343 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:06:54.895674 20343 webserver.cc:460] Webserver started at http://127.11.182.193:38043/ using document root <none> and password file <none>
I20260929 02:06:54.896366 20343 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:54.896471 20343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:54.896721 20343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:54.899803 20343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "443f84a896e644b384d33a176b7658a8"
format_stamp: "Formatted at 2026-09-29 02:06:54 on dist-test-slave-q06p"
server_key: "f411cc3d5a906fbcb3571100e6412c39"
server_key_iv: "cbd11fc4f2743c8bf2985ef4cb6e2a54"
server_key_version: "encryptionkey@0"
I20260929 02:06:54.900573 20343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "443f84a896e644b384d33a176b7658a8"
format_stamp: "Formatted at 2026-09-29 02:06:54 on dist-test-slave-q06p"
server_key: "f411cc3d5a906fbcb3571100e6412c39"
server_key_iv: "cbd11fc4f2743c8bf2985ef4cb6e2a54"
server_key_version: "encryptionkey@0"
I20260929 02:06:54.906225 20343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:06:54.909829 20358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:54.911427 20343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:06:54.911623 20343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "443f84a896e644b384d33a176b7658a8"
format_stamp: "Formatted at 2026-09-29 02:06:54 on dist-test-slave-q06p"
server_key: "f411cc3d5a906fbcb3571100e6412c39"
server_key_iv: "cbd11fc4f2743c8bf2985ef4cb6e2a54"
server_key_version: "encryptionkey@0"
I20260929 02:06:54.911854 20343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:54.925747 20343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:54.929957 20343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:54.930258 20343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:54.931308 20343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:54.932705 20343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:54.932778 20343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:54.932927 20343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:54.933013 20343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:54.966809 20343 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:34339
I20260929 02:06:54.966902 20470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:34339 every 8 connection(s)
I20260929 02:06:54.968160 20343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:54.973301 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 20343
I20260929 02:06:54.973527 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:06:54.974165 11995 external_mini_cluster.cc:1428] Setting key de3be61770ba4596997d3b2acc6b0613
Sep 29 02:06:54 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647614, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:54.991487 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:54.971057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:42077 (local address 127.11.182.254:33325)
0929 02:06:54.971534 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:54.971539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:54.972474 (+ 935us) server_negotiation.cc:407] Connection header received
0929 02:06:54.974521 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:54.974535 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:54.974675 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:54.974805 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:54.976941 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:54.977766 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:54.980387 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:54.980800 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:54.980907 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:54.985545 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:54.985551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:54.985554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:54.985587 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:54.988137 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:54.989128 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:54.989133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:54.989137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:54.989248 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:54.989742 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:54.989746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:54.989748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:54.990005 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:54.990111 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:54.991059 (+ 948us) server_negotiation.cc:299] Negotiation successful
0929 02:06:54.991231 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":250,"threads_started":1}
I20260929 02:06:54.993099 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:54.971587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:42077)
0929 02:06:54.972306 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:54.972352 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:54.974114 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:54.975350 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:54.975366 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:54.975946 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:54.976756 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:54.976787 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:54.977926 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:54.977935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:54.980221 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:54.980240 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:54.981052 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:54.981061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:54.981419 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:54.982171 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:54.982214 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:54.985279 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:54.988310 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:54.988323 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:54.988346 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:54.988990 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:54.989357 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:54.989365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:54.989371 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:54.989623 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:54.990138 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:54.990149 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:54.990633 (+ 484us) client_negotiation.cc:769] Sending connection context
0929 02:06:54.990969 (+ 336us) client_negotiation.cc:240] Negotiation successful
0929 02:06:54.991589 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":287,"threads_started":1}
I20260929 02:06:54.994755 20471 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33325
I20260929 02:06:54.995118 20471 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:54.995884 20471 heartbeater.cc:511] Master 127.11.182.254:33325 requested a full tablet report, sending...
I20260929 02:06:54.997977 20281 ts_manager.cc:194] Registered new tserver with Master: 443f84a896e644b384d33a176b7658a8 (127.11.182.193:34339)
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:06:54.999578 20281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:42077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:06:55.026551 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33325
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:55.360670 20477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:55.360957 20477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:55.361054 20477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:55.369207 20477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:55.369325 20477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:55.369439 20477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:55.379522 20477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:55.380755 20477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:55.382026 20477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:55.395234 20482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:55.395195 20485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:55.395256 20483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:55.395815 20477 server_base.cc:1034] running on GCE node
I20260929 02:06:55.396545 20477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:55.397542 20477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:55.398753 20477 hybrid_clock.cc:648] HybridClock initialized: now 1790647615398666 us; error 76 us; skew 500 ppm
I20260929 02:06:55.399117 20477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:55 dist-test-slave-q06p krb5kdc[20236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:55.402388 20477 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:06:55.403939 20477 webserver.cc:460] Webserver started at http://127.11.182.194:41119/ using document root <none> and password file <none>
I20260929 02:06:55.404661 20477 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:55.404816 20477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:55.405133 20477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:55.408227 20477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "51449467ebe0420f8aefa99bd0534e7e"
format_stamp: "Formatted at 2026-09-29 02:06:55 on dist-test-slave-q06p"
server_key: "c47c44c8214364a8be6062e02375f4dd"
server_key_iv: "956bfecd86dc5c08cfd5470cf0826087"
server_key_version: "encryptionkey@0"
I20260929 02:06:55.409013 20477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "51449467ebe0420f8aefa99bd0534e7e"
format_stamp: "Formatted at 2026-09-29 02:06:55 on dist-test-slave-q06p"
server_key: "c47c44c8214364a8be6062e02375f4dd"
server_key_iv: "956bfecd86dc5c08cfd5470cf0826087"
server_key_version: "encryptionkey@0"
I20260929 02:06:55.414223 20477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:06:55.417987 20492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:55.419570 20477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:55.419768 20477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "51449467ebe0420f8aefa99bd0534e7e"
format_stamp: "Formatted at 2026-09-29 02:06:55 on dist-test-slave-q06p"
server_key: "c47c44c8214364a8be6062e02375f4dd"
server_key_iv: "956bfecd86dc5c08cfd5470cf0826087"
server_key_version: "encryptionkey@0"
I20260929 02:06:55.419986 20477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:55.435377 20477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:55.440284 20477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:55.440594 20477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:55.441654 20477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:55.443125 20477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:55.443198 20477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:55.443368 20477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:55.443439 20477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:55.476438 20477 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:46619
I20260929 02:06:55.476619 20604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:46619 every 8 connection(s)
I20260929 02:06:55.477946 20477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:55.479507 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 20477
I20260929 02:06:55.479679 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:06:55.480113 11995 external_mini_cluster.cc:1428] Setting key ee566ee20b694e82944a48ca095fdef7
Sep 29 02:06:55 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:55.499522 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:55.480906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:59375 (local address 127.11.182.254:33325)
0929 02:06:55.481133 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:55.481138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:55.482515 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 02:06:55.484497 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:55.484503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:55.484592 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:55.484723 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:55.486308 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:55.487115 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:55.488720 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:55.489158 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:55.489297 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:55.493578 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:55.493585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:55.493589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:55.493632 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:55.496333 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:55.497135 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:55.497141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:55.497144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:55.497282 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:55.497729 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:55.497735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:55.497738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:55.498005 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:55.498166 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:55.498932 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 02:06:55.499161 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:06:55.500583 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:55.481521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:59375)
0929 02:06:55.482360 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:55.482412 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:55.484203 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:55.484968 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:55.484977 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:55.485524 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:55.486113 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:55.486131 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:55.487277 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:55.487281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:55.488552 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:55.488564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:55.489391 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:55.489399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:55.489630 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:55.490360 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:55.490382 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:55.493259 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:55.496516 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:55.496527 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:55.496553 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:55.497000 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:55.497399 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:55.497404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:55.497408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:55.497609 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:55.498141 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:55.498150 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:55.498527 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 02:06:55.498801 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 02:06:55.499423 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":356,"threads_started":1}
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:06:55.502348 20605 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33325
I20260929 02:06:55.502745 20605 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:55.503551 20605 heartbeater.cc:511] Master 127.11.182.254:33325 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
I20260929 02:06:55.505205 20281 ts_manager.cc:194] Registered new tserver with Master: 51449467ebe0420f8aefa99bd0534e7e (127.11.182.194:46619)
I20260929 02:06:55.506060 20281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:59375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:06:55.538594 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33325
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:55.884792 20610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:55.885111 20610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:55.885210 20610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:55.892720 20610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:55.892853 20610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:55.892969 20610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:55.903218 20610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:55.904589 20610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:55.905938 20610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:55.920389 20616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:55.920444 20618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:55.920475 20615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:55.921576 20610 server_base.cc:1034] running on GCE node
I20260929 02:06:55.922256 20610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:55.923527 20610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:55.924759 20610 hybrid_clock.cc:648] HybridClock initialized: now 1790647615924679 us; error 66 us; skew 500 ppm
I20260929 02:06:55.925154 20610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:55 dist-test-slave-q06p krb5kdc[20236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:55.928437 20610 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:06:55.929996 20610 webserver.cc:460] Webserver started at http://127.11.182.195:45391/ using document root <none> and password file <none>
I20260929 02:06:55.930751 20610 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:55.930922 20610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:55.931257 20610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:06:55.934356 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "88c5621aa1924104ae4e8ce02ad1516b"
format_stamp: "Formatted at 2026-09-29 02:06:55 on dist-test-slave-q06p"
server_key: "3ad57b5d794c2199cb444c8a46e6a650"
server_key_iv: "bcb5424bd20e9a8ed209f85b25830962"
server_key_version: "encryptionkey@0"
I20260929 02:06:55.935279 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "88c5621aa1924104ae4e8ce02ad1516b"
format_stamp: "Formatted at 2026-09-29 02:06:55 on dist-test-slave-q06p"
server_key: "3ad57b5d794c2199cb444c8a46e6a650"
server_key_iv: "bcb5424bd20e9a8ed209f85b25830962"
server_key_version: "encryptionkey@0"
I20260929 02:06:55.941074 20610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:06:55.945008 20625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:55.946580 20610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:06:55.946779 20610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "88c5621aa1924104ae4e8ce02ad1516b"
format_stamp: "Formatted at 2026-09-29 02:06:55 on dist-test-slave-q06p"
server_key: "3ad57b5d794c2199cb444c8a46e6a650"
server_key_iv: "bcb5424bd20e9a8ed209f85b25830962"
server_key_version: "encryptionkey@0"
I20260929 02:06:55.947065 20610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:55.966240 20610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:55.970803 20610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:55.971124 20610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:55.972079 20610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:55.973429 20610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:55.973501 20610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:55.973682 20610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:55.973732 20610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:56.002705 20471 heartbeater.cc:503] Master 127.11.182.254:33325 was elected leader, sending a full tablet report...
I20260929 02:06:56.008332 20610 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:46857
I20260929 02:06:56.008374 20737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:46857 every 8 connection(s)
I20260929 02:06:56.009660 20610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:56.011396 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 20610
I20260929 02:06:56.011592 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:06:56.012187 11995 external_mini_cluster.cc:1428] Setting key 10ff517753660bb3e16e66a06ccc8c7a
Sep 29 02:06:56 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:06:56.032249 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:56.012746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:46103 (local address 127.11.182.254:33325)
0929 02:06:56.013269 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:56.013275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:56.014926 (+ 1651us) server_negotiation.cc:407] Connection header received
0929 02:06:56.016567 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:56.016573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:56.016666 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:56.016781 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:56.018350 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.019248 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.020854 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.021269 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.021408 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:56.025418 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:56.025424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:56.025428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:56.025464 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:56.028715 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:56.029830 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:56.029835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:56.029837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:56.029959 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:56.030451 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:56.030455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:56.030458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:56.030767 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:56.030984 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:56.031735 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 02:06:56.031935 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":269,"threads_started":1}
I20260929 02:06:56.033262 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:56.013423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:46103)
0929 02:06:56.014309 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:56.014356 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:56.016278 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:56.017033 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:56.017041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:56.017588 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:56.018174 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:56.018191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.019410 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.019414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:56.020688 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:56.020700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.021517 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.021524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:56.021781 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:56.022475 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:56.022503 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:56.025135 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:06:56.028910 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:56.028919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:56.028937 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:56.029691 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:56.030084 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:06:56.030088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:06:56.030091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:06:56.030318 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:06:56.030975 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:06:56.030982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:06:56.031354 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 02:06:56.031619 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 02:06:56.032064 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":342,"threads_started":1}
I20260929 02:06:56.035022 20738 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33325
I20260929 02:06:56.035362 20738 heartbeater.cc:464] Registering TS with master...
I20260929 02:06:56.036141 20738 heartbeater.cc:511] Master 127.11.182.254:33325 requested a full tablet report, sending...
I20260929 02:06:56.037823 20281 ts_manager.cc:194] Registered new tserver with Master: 88c5621aa1924104ae4e8ce02ad1516b (127.11.182.195:46857)
I20260929 02:06:56.038785 20281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:46103
I20260929 02:06:56.041098 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab.
Entry for principal oryx/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab.
WARNING: no policy specified for oryx/127.11.182.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab.
Entry for principal oryx/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab.
WARNING: no policy specified for oryx/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab.
Entry for principal oryx/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab.
I20260929 02:06:56.186554 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 20343
Sep 29 02:06:56 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:06:56.203755 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:56.187868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.11.182.193:34339)
0929 02:06:56.188506 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:56.188513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:56.188561 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:06:56.188627 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:56.188633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:56.188844 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:56.188974 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:56.190376 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.191608 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.192874 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.193554 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.193678 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:56.197899 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:56.197912 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:56.197936 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:56.197979 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:56.200877 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:56.201682 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:56.201689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:56.201691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:56.201822 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:56.202188 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:56.202191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:56.202193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:56.202535 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:56.202779 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:56.203111 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 02:06:56.203309 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":330,"threads_started":1}
I20260929 02:06:56.205068 20466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:56.289579 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 20343
I20260929 02:06:56.307605 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 20477
Sep 29 02:06:56 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:06:56.327196 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:56.308672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.11.182.194:46619)
0929 02:06:56.309180 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:56.309187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:56.309212 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:06:56.309270 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:56.309276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:56.309484 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:56.309619 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:56.310902 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.311927 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.313168 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.313974 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.314110 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:56.319755 (+ 5645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:56.319776 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:56.319799 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:56.319862 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:56.323886 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:56.324908 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:56.324916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:56.324919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:56.325069 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:56.325403 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:56.325406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:56.325409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:56.325816 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:56.326077 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:56.326439 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 02:06:56.326753 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":263,"threads_started":1}
I20260929 02:06:56.328547 20600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:56.414767 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 20477
I20260929 02:06:56.431834 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 20610
Sep 29 02:06:56 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:06:56.451776 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:56.432958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.11.182.195:46857)
0929 02:06:56.433900 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:56.433908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:56.433948 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:06:56.434016 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:56.434023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:56.434250 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:56.434411 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:56.436521 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.437639 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.439064 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:56.439654 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:56.439820 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:56.444925 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:56.444940 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:56.444957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:56.445006 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:56.448488 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:56.449357 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:56.449366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:56.449368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:56.449554 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:56.450018 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:56.450022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:56.450024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:56.450391 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:56.450662 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:56.451122 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 02:06:56.451382 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":509,"threads_started":1}
I20260929 02:06:56.453181 20733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:56.540266 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 20610
I20260929 02:06:56.556622 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 20249
I20260929 02:06:56.558068 20311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:06:56.665859 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 20249
I20260929 02:06:56.688063 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33325
--webserver_interface=127.11.182.254
--webserver_port=40741
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:33325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:06:57.047465 20756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:57.047832 20756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:57.047999 20756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:57.055392 20756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:06:57.055557 20756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:57.055668 20756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:57.055775 20756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:06:57.055862 20756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:06:57.065848 20756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:33325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=40741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:57.067269 20756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:57.068596 20756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:57.080559 20761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:57.080633 20764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:57.080675 20762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:57.081676 20756 server_base.cc:1034] running on GCE node
I20260929 02:06:57.082505 20756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:57.084031 20756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:57.085229 20756 hybrid_clock.cc:648] HybridClock initialized: now 1790647617085163 us; error 57 us; skew 500 ppm
I20260929 02:06:57.085583 20756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:57 dist-test-slave-q06p krb5kdc[20236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647617, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:57.088774 20756 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:06:57.090261 20756 webserver.cc:460] Webserver started at http://127.11.182.254:40741/ using document root <none> and password file <none>
I20260929 02:06:57.090986 20756 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:57.091101 20756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:57.096376 20756 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:06:57.099262 20771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:57.100593 20756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:57.100795 20756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "d7afda6ccc4b4308bd0075eb1df7609b"
format_stamp: "Formatted at 2026-09-29 02:06:54 on dist-test-slave-q06p"
server_key: "fd6857a7164d5a871ef53eaf7c64f0ce"
server_key_iv: "50c6b0d2ea745e58d8481c8149909d1f"
server_key_version: "encryptionkey@0"
I20260929 02:06:57.101763 20756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:57.144855 20756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:57.149296 20756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:57.149595 20756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:57.167887 20756 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:33325
I20260929 02:06:57.167908 20822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:33325 every 8 connection(s)
I20260929 02:06:57.169404 20756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:06:57.173280 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 20756
I20260929 02:06:57.174319 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:34339
--local_ip_for_outbound_sockets=127.11.182.193
--tserver_master_addrs=127.11.182.254:33325
--webserver_port=38043
--webserver_interface=127.11.182.193
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.182.193
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:06:57.178280 20823 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:06:57.183516 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: Bootstrap starting.
I20260929 02:06:57.203260 20823 log.cc:824] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:06:57.213330 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:06:57.213958 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: Bootstrap complete.
I20260929 02:06:57.223995 20823 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } }
I20260929 02:06:57.224918 20823 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7afda6ccc4b4308bd0075eb1df7609b, State: Initialized, Role: FOLLOWER
I20260929 02:06:57.225687 20823 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } }
I20260929 02:06:57.225957 20823 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:06:57.226094 20823 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:06:57.226311 20823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:06:57.230579 20823 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } }
I20260929 02:06:57.231194 20823 leader_election.cc:302] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7afda6ccc4b4308bd0075eb1df7609b; no voters:
I20260929 02:06:57.231911 20823 leader_election.cc:288] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:06:57.232076 20827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:06:57.234319 20827 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [term 2 LEADER]: Becoming Leader. State: Replica: d7afda6ccc4b4308bd0075eb1df7609b, State: Running, Role: LEADER
I20260929 02:06:57.234547 20823 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:06:57.235001 20827 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } }
I20260929 02:06:57.238880 20829 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7afda6ccc4b4308bd0075eb1df7609b. Latest consensus state: current_term: 2 leader_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } } }
I20260929 02:06:57.239425 20829 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:57.243361 20828 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7afda6ccc4b4308bd0075eb1df7609b" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33325 } } }
I20260929 02:06:57.243687 20828 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b [sys.catalog]: This master's current role is: LEADER
I20260929 02:06:57.244684 20835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:06:57.251014 20835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:06:57.252573 20835 catalog_manager.cc:1259] Loaded cluster ID: c05cb5dc8c3e463f8a4c40f9c9753035
I20260929 02:06:57.252763 20835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:06:57.256881 20835 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:06:57.259289 20835 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7afda6ccc4b4308bd0075eb1df7609b: Loaded TSK: 0
I20260929 02:06:57.261035 20835 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:06:57.526106 20825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:57.526408 20825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:57.526557 20825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:57.534032 20825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:57.534205 20825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:57.534389 20825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:06:57.544957 20825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.193.keytab
--principal=oryx/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:34339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=38043
--tserver_master_addrs=127.11.182.254:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:57.546355 20825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:57.547827 20825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:57.562569 20850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:57.562626 20849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:57.563561 20825 server_base.cc:1034] running on GCE node
W20260929 02:06:57.563930 20852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:57.564745 20825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:57.566010 20825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:57.567260 20825 hybrid_clock.cc:648] HybridClock initialized: now 1790647617567175 us; error 62 us; skew 500 ppm
I20260929 02:06:57.567626 20825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:57 dist-test-slave-q06p krb5kdc[20236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647617, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:57.571758 20825 init.cc:377] Logged in from keytab as oryx/127.11.182.193@KRBTEST.COM (short username oryx)
I20260929 02:06:57.573701 20825 webserver.cc:460] Webserver started at http://127.11.182.193:38043/ using document root <none> and password file <none>
I20260929 02:06:57.574430 20825 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:57.574589 20825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:57.580332 20825 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 02:06:57.583218 20859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:57.584722 20825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:06:57.584903 20825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "443f84a896e644b384d33a176b7658a8"
format_stamp: "Formatted at 2026-09-29 02:06:54 on dist-test-slave-q06p"
server_key: "f411cc3d5a906fbcb3571100e6412c39"
server_key_iv: "cbd11fc4f2743c8bf2985ef4cb6e2a54"
server_key_version: "encryptionkey@0"
I20260929 02:06:57.585873 20825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:57.610332 20825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:57.614669 20825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:57.614998 20825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:57.616034 20825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:57.617398 20825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:57.617527 20825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:57.617657 20825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:57.617749 20825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:57.652017 20825 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:34339
I20260929 02:06:57.652139 20972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:34339 every 8 connection(s)
I20260929 02:06:57.653486 20825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:06:57.659417 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 20825
I20260929 02:06:57.660060 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:46619
--local_ip_for_outbound_sockets=127.11.182.194
--tserver_master_addrs=127.11.182.254:33325
--webserver_port=41119
--webserver_interface=127.11.182.194
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.182.194
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:06:57 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:57 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:57.675277 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:57.656747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:45859)
0929 02:06:57.658133 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:57.658179 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:57.660173 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:57.662034 (+ 1861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:57.662042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:57.662588 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:57.663244 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:57.663263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.665629 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.665633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:57.667268 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:57.667286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.668401 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.668411 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:57.668746 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:57.669534 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:57.669574 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:57.673897 (+ 4323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":803,"threads_started":1}
W20260929 02:06:57.675341 20976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:57.656693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:45859 (local address 127.11.182.254:33325)
0929 02:06:57.657480 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:57.657492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:57.658328 (+ 836us) server_negotiation.cc:407] Connection header received
0929 02:06:57.660864 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:57.660886 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:57.661417 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:57.661843 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:57.663452 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.665433 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.667449 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.668144 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.668289 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:57.674267 (+ 5978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:45859: BlockingRecv error: recv got EOF from 127.11.182.193:45859 (error 108)
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":334,"threads_started":1}
W20260929 02:06:57.676098 20973 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:33325 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.182.254:33325: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Sep 29 02:06:57 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:57 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:57.686491 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:57.676781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:42653)
0929 02:06:57.677010 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:57.677032 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:57.677157 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:57.677493 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:57.677500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:57.677987 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:57.678207 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:57.678222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.679419 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.679428 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:57.681183 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:57.681200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.681926 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.681934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:57.682155 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:57.682780 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:57.682809 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:57.686194 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 02:06:57.686635 20976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:57.676912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:42653 (local address 127.11.182.254:33325)
0929 02:06:57.677060 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:57.677064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:57.677082 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:06:57.677285 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:57.677288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:57.677346 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:57.677427 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:57.678373 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.679262 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.681345 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.681768 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.681868 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:57.686409 (+ 4541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:42653: BlockingRecv error: recv got EOF from 127.11.182.193:42653 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:06:57 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:57 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:57.695380 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:57.687323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:58901)
0929 02:06:57.687477 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:57.687495 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:57.687587 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:57.687900 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:57.687903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:57.688156 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:57.688310 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:57.688317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.689707 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.689711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:57.690826 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:57.690837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.691800 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.691804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:57.691921 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:57.692513 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:57.692529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:57.695161 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:06:57.695700 20976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:57.687394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:58901 (local address 127.11.182.254:33325)
0929 02:06:57.687565 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:57.687571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:57.687590 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:06:57.687685 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:57.687690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:57.687772 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:57.687872 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:57.688435 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.689570 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.691004 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:57.691663 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:57.691779 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:57.695404 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:58901: BlockingRecv error: recv got EOF from 127.11.182.193:58901 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 02:06:57.695824 20973 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:06:57.995431 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:57.995712 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:57.995873 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:58.003309 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:58.003423 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:58.003600 20977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:06:58.013760 20977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.194.keytab
--principal=oryx/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:46619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=41119
--tserver_master_addrs=127.11.182.254:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:58.015192 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:58.016551 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:58.030433 20982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:58.030476 20985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:58.030515 20983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:58.031919 20977 server_base.cc:1034] running on GCE node
I20260929 02:06:58.032610 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:58.033711 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:58.034993 20977 hybrid_clock.cc:648] HybridClock initialized: now 1790647618034933 us; error 38 us; skew 500 ppm
I20260929 02:06:58.035368 20977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647618, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:58.038964 20977 init.cc:377] Logged in from keytab as oryx/127.11.182.194@KRBTEST.COM (short username oryx)
I20260929 02:06:58.040575 20977 webserver.cc:460] Webserver started at http://127.11.182.194:41119/ using document root <none> and password file <none>
I20260929 02:06:58.041285 20977 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:58.041410 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:58.047756 20977 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:58.050634 20992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:58.052229 20977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:06:58.052382 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "51449467ebe0420f8aefa99bd0534e7e"
format_stamp: "Formatted at 2026-09-29 02:06:55 on dist-test-slave-q06p"
server_key: "c47c44c8214364a8be6062e02375f4dd"
server_key_iv: "956bfecd86dc5c08cfd5470cf0826087"
server_key_version: "encryptionkey@0"
I20260929 02:06:58.053341 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:58.080054 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:58.084688 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:58.084928 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:58.085919 20977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:58.087282 20977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:58.087368 20977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:58.087451 20977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:58.087517 20977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:58.121920 20977 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:46619
I20260929 02:06:58.121995 21104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:46619 every 8 connection(s)
I20260929 02:06:58.123431 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:06:58.125455 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 20977
I20260929 02:06:58.126199 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:46857
--local_ip_for_outbound_sockets=127.11.182.195
--tserver_master_addrs=127.11.182.254:33325
--webserver_port=45391
--webserver_interface=127.11.182.195
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.182.195
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:06:58.141979 20976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:58.126133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:60315 (local address 127.11.182.254:33325)
0929 02:06:58.126347 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:58.126355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:58.127955 (+ 1600us) server_negotiation.cc:407] Connection header received
0929 02:06:58.129972 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:58.129979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:58.130105 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:58.130230 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:58.131969 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.132715 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.135401 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.135903 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.136065 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.141735 (+ 5670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:60315: BlockingRecv error: recv got EOF from 127.11.182.194:60315 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:06:58.142757 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:58.126876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:60315)
0929 02:06:58.127756 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:58.127804 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:58.129649 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:58.130500 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:58.130510 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:58.131152 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:58.131769 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.131788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.132885 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.132897 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.135239 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.135260 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.136168 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.136178 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.136542 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.137305 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:58.137344 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:58.141553 (+ 4209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":342,"threads_started":1}
W20260929 02:06:58.143605 21105 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:33325 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.182.254:33325: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:58.153998 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:58.144286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:51287)
0929 02:06:58.144467 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:58.144503 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:58.144620 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:58.145119 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:58.145123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:58.145452 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:58.145588 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.145596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.146772 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.146782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.148598 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.148620 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.149319 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.149329 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.149593 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.150229 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:58.150260 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:58.153699 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:06:58.154347 20976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:58.144482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:51287 (local address 127.11.182.254:33325)
0929 02:06:58.144795 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:58.144821 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:58.144842 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:06:58.144902 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:58.144905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:58.144982 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:58.145098 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:58.145744 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.146541 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.148763 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.149146 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.149250 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.153945 (+ 4695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:51287: BlockingRecv error: recv got EOF from 127.11.182.194:51287 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:58.163591 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:58.154832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:37873)
0929 02:06:58.155006 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:58.155027 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:58.155133 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:58.155673 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:58.155676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:58.155932 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:58.156093 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.156102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.157634 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.157639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.158700 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.158714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.159741 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.159745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.159869 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.160496 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:58.160513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:58.163365 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 02:06:58.163906 20976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:58.155150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:37873 (local address 127.11.182.254:33325)
0929 02:06:58.155343 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:58.155350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:58.155371 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:06:58.155434 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:58.155440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:58.155525 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:58.155633 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:58.156228 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.157470 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.158876 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.159586 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.159743 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.163662 (+ 3919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:37873: BlockingRecv error: recv got EOF from 127.11.182.194:37873 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 02:06:58.164168 21105 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:06:58.473795 21108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:06:58.474076 21108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:06:58.474169 21108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:06:58.481438 21108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:06:58.481554 21108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:06:58.481674 21108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:06:58.491923 21108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/krb5kdc/oryx_127.11.182.195.keytab
--principal=oryx/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:46857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=45391
--tserver_master_addrs=127.11.182.254:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:06:58.493216 21108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:06:58.494452 21108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:06:58.507679 21113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:58.507740 21114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:06:58.507774 21116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:06:58.508471 21108 server_base.cc:1034] running on GCE node
I20260929 02:06:58.509156 21108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:06:58.510197 21108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:06:58.511428 21108 hybrid_clock.cc:648] HybridClock initialized: now 1790647618511378 us; error 52 us; skew 500 ppm
I20260929 02:06:58.511754 21108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647618, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:06:58.514993 21108 init.cc:377] Logged in from keytab as oryx/127.11.182.195@KRBTEST.COM (short username oryx)
I20260929 02:06:58.516521 21108 webserver.cc:460] Webserver started at http://127.11.182.195:45391/ using document root <none> and password file <none>
I20260929 02:06:58.517230 21108 fs_manager.cc:359] Metadata directory not provided
I20260929 02:06:58.517388 21108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:06:58.523108 21108 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:06:58.525882 21123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:58.527418 21108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:06:58.527603 21108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "88c5621aa1924104ae4e8ce02ad1516b"
format_stamp: "Formatted at 2026-09-29 02:06:55 on dist-test-slave-q06p"
server_key: "3ad57b5d794c2199cb444c8a46e6a650"
server_key_iv: "bcb5424bd20e9a8ed209f85b25830962"
server_key_version: "encryptionkey@0"
I20260929 02:06:58.528580 21108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:06:58.564574 21108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:06:58.569146 21108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:06:58.569460 21108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:06:58.570466 21108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:06:58.571988 21108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:06:58.572075 21108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:58.572152 21108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:06:58.572207 21108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:06:58.606794 21108 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:46857
I20260929 02:06:58.606900 21235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:46857 every 8 connection(s)
I20260929 02:06:58.608354 21108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:06:58.611616 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 21108
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:06:58.625365 20976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:58.610770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:54807 (local address 127.11.182.254:33325)
0929 02:06:58.611021 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:58.611027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:58.612474 (+ 1447us) server_negotiation.cc:407] Connection header received
0929 02:06:58.614330 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:58.614335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:58.614427 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:58.614536 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:58.616365 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.617096 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.618627 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.619140 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.619288 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.625066 (+ 5778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:54807: BlockingRecv error: recv got EOF from 127.11.182.195:54807 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 02:06:58.626432 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:58.611455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:54807)
0929 02:06:58.612326 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:58.612371 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:58.614064 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:58.614784 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:58.614792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:58.615618 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:58.616189 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.616206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.617252 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.617263 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.618471 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.618482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.619378 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.619384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.619701 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.620441 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:58.620482 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:58.624877 (+ 4395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":340,"threads_started":1}
W20260929 02:06:58.627317 21236 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:33325 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.182.254:33325: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:58.635535 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:58.612575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.11.182.254:33325)
0929 02:06:58.619259 (+ 6684us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:58.619263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:58.619278 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:06:58.619319 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:58.619322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:58.619381 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:58.619467 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:58.621490 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.622571 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.624579 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.625148 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.625290 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.628509 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:06:58.628521 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:06:58.628544 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:06:58.628594 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:06:58.631956 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:58.632781 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:58.632788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:58.632795 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:58.632964 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:06:58.633382 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:06:58.633385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:06:58.633387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:06:58.633921 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:06:58.634145 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:06:58.634888 (+ 743us) server_negotiation.cc:299] Negotiation successful
0929 02:06:58.635252 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":6578,"thread_start_us":6469,"threads_started":1}
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:58.636736 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:58.627937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:36677)
0929 02:06:58.628140 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:58.628160 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:58.628290 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:58.628621 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:58.628624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:58.628904 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:58.629033 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.629040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.630511 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.630517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.631492 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.631503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.632420 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.632427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.632614 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.633442 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:58.633459 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:58.636537 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 02:06:58.637405 20976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:58.628026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:36677 (local address 127.11.182.254:33325)
0929 02:06:58.628175 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:58.628179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:58.628242 (+ 63us) server_negotiation.cc:407] Connection header received
0929 02:06:58.628406 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:58.628410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:58.628492 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:58.628590 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:58.629177 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.630346 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.631630 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.632284 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.632374 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.636755 (+ 4381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:36677: BlockingRecv error: recv got EOF from 127.11.182.195:36677 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:58.645735 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:58.637490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:39203)
0929 02:06:58.638349 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:58.638365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:58.638455 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:58.638825 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:58.638828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:58.639278 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:58.639474 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.639488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.640428 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.640432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.641342 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.641352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.641969 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.641973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.642090 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.642690 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:58.642708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:58.645434 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":764}
W20260929 02:06:58.646134 21236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:06:58.646150 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:58.637841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:39203 (local address 127.11.182.254:33325)
0929 02:06:58.638183 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:58.638189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:58.638437 (+ 248us) server_negotiation.cc:407] Connection header received
0929 02:06:58.638573 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:58.638577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:58.638687 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:58.638790 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:58.639648 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.640309 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.641457 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.641851 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.641944 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.645780 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:39203: BlockingRecv error: recv got EOF from 127.11.182.195:39203 (error 108)
Metrics: {"server-negotiator.queue_time_us":173}
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:58 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:58.706084 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:58.696404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:48833)
0929 02:06:58.696896 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:58.696913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:58.697028 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:58.697322 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:58.697325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:58.697762 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:58.697945 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.697953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.698957 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.698966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.700667 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:58.700686 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.701770 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.701778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:58.701976 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.702567 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:58.702584 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:58.705746 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":278,"threads_started":1}
W20260929 02:06:58.706375 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:58.696577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:48833 (local address 127.11.182.254:33325)
0929 02:06:58.696822 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:58.696828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:58.696988 (+ 160us) server_negotiation.cc:407] Connection header received
0929 02:06:58.697126 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:58.697130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:58.697191 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:58.697270 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:58.698060 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.698770 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.700871 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:58.701516 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:58.701744 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:58.705951 (+ 4207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:48833: BlockingRecv error: recv got EOF from 127.11.182.193:48833 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 02:06:59 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:59 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:59.173517 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:59.164715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:46447)
0929 02:06:59.165254 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:59.165270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:59.165379 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:59.165704 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:59.165708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:59.166134 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:59.166320 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:59.166329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.167545 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.167550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:59.168512 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:59.168525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.169295 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.169301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:59.169492 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:59.170136 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:59.170154 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:59.173173 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":294,"threads_started":1}
W20260929 02:06:59.173607 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:59.164758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:46447 (local address 127.11.182.254:33325)
0929 02:06:59.164908 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:59.164913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:59.165339 (+ 426us) server_negotiation.cc:407] Connection header received
0929 02:06:59.165473 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:59.165476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:59.165546 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:59.165628 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:59.166460 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.167381 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.168660 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.169104 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.169241 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:59.173386 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:46447: BlockingRecv error: recv got EOF from 127.11.182.194:46447 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 02:06:59 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:59 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:59.664291 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:59.647233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:39167)
0929 02:06:59.647827 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:59.647843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:59.648014 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:59.648611 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:59.648615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:59.649283 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:59.649559 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:59.649573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.650804 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.650808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:59.651923 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:59.651936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.652975 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.652980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:59.653113 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:59.659641 (+ 6528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:59.659689 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:59.663945 (+ 4256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":409,"threads_started":1}
W20260929 02:06:59.664451 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:59.647233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:39167 (local address 127.11.182.254:33325)
0929 02:06:59.647513 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:59.647522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:59.647963 (+ 441us) server_negotiation.cc:407] Connection header received
0929 02:06:59.648228 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:59.648234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:59.648319 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:59.648393 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:59.649736 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.650649 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.652097 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.652760 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.652902 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:59.664180 (+ 11278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:39167: BlockingRecv error: recv got EOF from 127.11.182.195:39167 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 02:06:59 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:06:59 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:06:59.716003 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:06:59.706982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:35959)
0929 02:06:59.707543 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 02:06:59.707559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:06:59.707702 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:06:59.708098 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:06:59.708103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:06:59.708512 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:06:59.708714 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:59.708723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.709741 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.709745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:59.710665 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:06:59.710677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.711644 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.711649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:06:59.711787 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:59.712464 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:06:59.712496 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:06:59.715653 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":327,"threads_started":1}
W20260929 02:06:59.716080 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:06:59.707106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:35959 (local address 127.11.182.254:33325)
0929 02:06:59.707286 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:06:59.707291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:06:59.707641 (+ 350us) server_negotiation.cc:407] Connection header received
0929 02:06:59.707842 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:06:59.707847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:06:59.707912 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:06:59.708008 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:06:59.708862 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.709587 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.710832 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:06:59.711404 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:06:59.711557 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:06:59.715866 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:35959: BlockingRecv error: recv got EOF from 127.11.182.193:35959 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:07:00 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:00 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:00.183321 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:00.174673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:42717)
0929 02:07:00.175250 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:00.175266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:00.175432 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:00.175815 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:00.175820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:00.176259 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:00.176435 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:00.176443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.177495 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.177501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:00.178409 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:00.178420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.179220 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.179226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:00.179405 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:00.180086 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:00.180108 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:00.183081 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":351,"threads_started":1}
W20260929 02:07:00.183651 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:00.174785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:42717 (local address 127.11.182.254:33325)
0929 02:07:00.175077 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:00.175083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:00.175362 (+ 279us) server_negotiation.cc:407] Connection header received
0929 02:07:00.175613 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:00.175618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:00.175678 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:00.175792 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:00.176590 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.177359 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.178560 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.179049 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.179188 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:00.183319 (+ 4131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:42717: BlockingRecv error: recv got EOF from 127.11.182.194:42717 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 02:07:00 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:00 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:00.674501 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:00.665782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:39237)
0929 02:07:00.666467 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:00.666486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:00.666615 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:00.666989 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:00.666993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:00.667404 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:00.667577 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:00.667586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.668595 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.668599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:00.669611 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:00.669624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.670315 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.670319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:00.670425 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:00.671133 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:00.671149 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:00.674257 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":445,"threads_started":1}
W20260929 02:07:00.674693 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:00.666116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:39237 (local address 127.11.182.254:33325)
0929 02:07:00.666395 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:00.666402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:00.666571 (+ 169us) server_negotiation.cc:407] Connection header received
0929 02:07:00.666725 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:00.666729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:00.666792 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:00.666950 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:00.667727 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.668458 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.669754 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.670180 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.670274 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:00.674458 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:39237: BlockingRecv error: recv got EOF from 127.11.182.195:39237 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 02:07:00 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:00 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:00.726051 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:00.717027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:37559)
0929 02:07:00.717593 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:00.717609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:00.717778 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:00.718124 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:00.718129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:00.718507 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:00.718691 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:00.718713 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.719871 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.719878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:00.721055 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:00.721068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.721789 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.721796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:00.721985 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:00.722651 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:00.722669 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:00.725783 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":293,"threads_started":1}
W20260929 02:07:00.726289 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:00.717159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:37559 (local address 127.11.182.254:33325)
0929 02:07:00.717407 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:00.717415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:00.717710 (+ 295us) server_negotiation.cc:407] Connection header received
0929 02:07:00.717911 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:00.717919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:00.717984 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:00.718094 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:00.718888 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.719698 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.721191 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:00.721628 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:00.721767 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:00.726042 (+ 4275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:37559: BlockingRecv error: recv got EOF from 127.11.182.193:37559 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 02:07:01 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:01 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:01.192899 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:01.184291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:59481)
0929 02:07:01.184825 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:01.184842 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:01.185002 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:01.185340 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:01.185345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:01.185716 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:01.185891 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:01.185899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.186981 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.186988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:01.187893 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:01.187904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.188639 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.188645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:01.188824 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:01.189525 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:01.189543 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:01.192625 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":286,"threads_started":1}
W20260929 02:07:01.193076 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:01.184377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:59481 (local address 127.11.182.254:33325)
0929 02:07:01.184642 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:01.184651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:01.184927 (+ 276us) server_negotiation.cc:407] Connection header received
0929 02:07:01.185138 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:01.185147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:01.185207 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:01.185310 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:01.186044 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.186784 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.188040 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.188478 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.188617 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:01.192836 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:59481: BlockingRecv error: recv got EOF from 127.11.182.194:59481 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 29 02:07:01 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:01 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:07:01.686698 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:01.675777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:41033 (local address 127.11.182.254:33325)
0929 02:07:01.676030 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:01.676040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:01.676455 (+ 415us) server_negotiation.cc:407] Connection header received
0929 02:07:01.676669 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:01.676678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:01.676779 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:01.676910 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:01.677664 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.679057 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.680707 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.681348 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.681514 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:01.686471 (+ 4957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:41033: BlockingRecv error: recv got EOF from 127.11.182.195:41033 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:07:01.686749 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:01.675621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:41033)
0929 02:07:01.676336 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:01.676355 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:01.676484 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:01.676970 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:01.676974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:01.677315 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:01.677490 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:01.677498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.679256 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.679261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:01.680544 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:01.680562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.681524 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.681533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:01.681791 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:01.682509 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:01.682540 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:01.686260 (+ 3720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":429,"threads_started":1}
Sep 29 02:07:01 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:01 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:01.738699 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:01.727190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:44969)
0929 02:07:01.727792 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:01.727816 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:01.727934 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:01.728365 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:01.728369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:01.728740 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:01.728890 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:01.728899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.730769 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.730779 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:01.732810 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:01.732834 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.733757 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.733767 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:01.734010 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:01.734818 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:01.734888 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:01.738464 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":365,"threads_started":1}
W20260929 02:07:01.738891 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:01.727290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:44969 (local address 127.11.182.254:33325)
0929 02:07:01.727530 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:01.727539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:01.727904 (+ 365us) server_negotiation.cc:407] Connection header received
0929 02:07:01.728064 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:01.728072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:01.728171 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:01.728302 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:01.729051 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.730522 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.733005 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:01.733566 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:01.733674 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:01.738677 (+ 5003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:44969: BlockingRecv error: recv got EOF from 127.11.182.193:44969 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 02:07:02 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:02 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:02.202495 21251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:02.193844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:39753)
0929 02:07:02.194381 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:02.194396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:02.194508 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:02.194889 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:02.194899 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:02.195281 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:02.195431 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:02.195439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.196508 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.196512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:02.197518 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:02.197530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.198234 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.198238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:02.198360 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:02.199003 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:02.199020 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:02.202156 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":321,"threads_started":1}
W20260929 02:07:02.202625 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:02.193954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:39753 (local address 127.11.182.254:33325)
0929 02:07:02.194119 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:02.194124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:02.194471 (+ 347us) server_negotiation.cc:407] Connection header received
0929 02:07:02.194621 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:02.194625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:02.194705 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:02.194789 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:02.195587 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.196368 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.197663 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.198083 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.198186 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:02.202342 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:39753: BlockingRecv error: recv got EOF from 127.11.182.194:39753 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:07:02 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:02 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:02.696557 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:02.687734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:48391)
0929 02:07:02.688266 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:02.688288 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:02.688491 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:02.688929 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:02.688934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:02.689498 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:02.689662 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:02.689672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.690765 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.690770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:02.691794 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:02.691807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.692551 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.692555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:02.692708 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:02.693471 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:02.693490 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:02.696324 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":286,"threads_started":1}
W20260929 02:07:02.696971 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:02.687813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:48391 (local address 127.11.182.254:33325)
0929 02:07:02.688019 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:02.688024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:02.688398 (+ 374us) server_negotiation.cc:407] Connection header received
0929 02:07:02.688666 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:02.688674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:02.688761 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:02.688859 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:02.689848 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.690605 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.691983 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.692397 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.692507 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:02.696659 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:48391: BlockingRecv error: recv got EOF from 127.11.182.195:48391 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 02:07:02 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:02 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:02.747763 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:02.739673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:52243)
0929 02:07:02.740128 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:02.740145 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:02.740253 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:02.740552 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:02.740555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:02.740908 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:02.741064 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:02.741072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.742063 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.742068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:02.743141 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:02.743153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.743858 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.743863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:02.743989 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:02.744675 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:02.744693 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:02.747502 (+ 2809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":223,"threads_started":1}
W20260929 02:07:02.748075 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:02.739714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:52243 (local address 127.11.182.254:33325)
0929 02:07:02.739930 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:02.739937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:02.740269 (+ 332us) server_negotiation.cc:407] Connection header received
0929 02:07:02.740350 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:02.740353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:02.740416 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:02.740505 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:02.741197 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.741916 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.743278 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:02.743729 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:02.743826 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:02.747783 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:52243: BlockingRecv error: recv got EOF from 127.11.182.193:52243 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:07:03 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:03 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:03.212381 21254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:03.203648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:38231)
0929 02:07:03.204304 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:03.204323 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:03.204443 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:03.204739 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:03.204743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:03.205145 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:03.205312 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:03.205322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.206483 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.206488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:03.207553 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:03.207568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.208251 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.208256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:03.208391 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:03.208997 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:03.209019 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:03.212010 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":357,"threads_started":1}
W20260929 02:07:03.212404 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:03.203811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:38231 (local address 127.11.182.254:33325)
0929 02:07:03.203977 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:03.203982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:03.204399 (+ 417us) server_negotiation.cc:407] Connection header received
0929 02:07:03.204547 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:03.204550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:03.204609 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:03.204710 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:03.205486 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.206334 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.207695 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.208114 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.208214 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:03.212202 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:38231: BlockingRecv error: recv got EOF from 127.11.182.194:38231 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:07:03 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:03 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:03.705368 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:03.697542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:57777)
0929 02:07:03.698073 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:03.698089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:03.698194 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:03.698489 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:03.698493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:03.698927 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:03.699145 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:03.699156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.700120 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.700124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:03.701079 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:03.701091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.701736 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.701740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:03.701852 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:03.702414 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:03.702431 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:03.705110 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":282,"threads_started":1}
W20260929 02:07:03.705790 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:03.697598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:57777 (local address 127.11.182.254:33325)
0929 02:07:03.697780 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:03.697784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:03.698158 (+ 374us) server_negotiation.cc:407] Connection header received
0929 02:07:03.698286 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:03.698290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:03.698345 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:03.698426 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:03.699280 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.699972 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.701203 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.701609 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.701705 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:03.705412 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:57777: BlockingRecv error: recv got EOF from 127.11.182.195:57777 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:07:03 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:03 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:07:03.758085 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:03.748938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:43931 (local address 127.11.182.254:33325)
0929 02:07:03.749195 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:03.749203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:03.749524 (+ 321us) server_negotiation.cc:407] Connection header received
0929 02:07:03.749750 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:03.749756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:03.749835 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:03.749950 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:03.750750 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.751629 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.752936 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.753438 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.753588 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:03.757904 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:43931: BlockingRecv error: recv got EOF from 127.11.182.193:43931 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:07:03.758526 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:03.748780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:43931)
0929 02:07:03.749371 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:03.749388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:03.749609 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:03.749986 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:03.749992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:03.750415 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:03.750589 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:03.750606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.751778 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.751786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:03.752779 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:03.752789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:03.753607 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:03.753613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:03.753802 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:03.754422 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:03.754440 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:03.758320 (+ 3880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":320,"threads_started":1}
Sep 29 02:07:04 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:04 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:04.221877 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:04.213334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:60687)
0929 02:07:04.213808 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:04.213825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:04.213985 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:04.214352 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:04.214355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:04.214934 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:04.215082 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:04.215092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.216437 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.216441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:04.217335 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:04.217346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.218003 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.218007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:04.218122 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:04.218765 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:04.218782 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:04.221559 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
W20260929 02:07:04.222050 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:04.213493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:60687 (local address 127.11.182.254:33325)
0929 02:07:04.213750 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:04.213756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:04.213913 (+ 157us) server_negotiation.cc:407] Connection header received
0929 02:07:04.214113 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:04.214125 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:04.214186 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:04.214297 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:04.215257 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.216261 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.217473 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.217876 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.217966 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:04.221776 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:60687: BlockingRecv error: recv got EOF from 127.11.182.194:60687 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 02:07:04 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:04 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:04.722677 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:04.706451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:41475)
0929 02:07:04.706992 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:04.707009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:04.707114 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:04.707397 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:04.707400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:04.707782 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:04.707933 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:04.707941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.708861 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.708864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:04.709730 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:04.709740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.710435 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.710441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:04.710570 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:04.719602 (+ 9032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:04.719625 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:04.722459 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":304,"threads_started":1}
W20260929 02:07:04.722883 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:04.706462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:41475 (local address 127.11.182.254:33325)
0929 02:07:04.706653 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:04.706658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:04.707078 (+ 420us) server_negotiation.cc:407] Connection header received
0929 02:07:04.707216 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:04.707219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:04.707277 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:04.707355 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:04.708050 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.708746 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.709851 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.710262 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.710356 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:04.722650 (+ 12294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:41475: BlockingRecv error: recv got EOF from 127.11.182.195:41475 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 02:07:04 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:04 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:04.768400 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:04.759543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:42595)
0929 02:07:04.760196 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:04.760213 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:04.760343 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:04.760665 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:04.760669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:04.761116 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:04.761293 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:04.761303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.762277 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.762281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:04.763365 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:04.763377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.764221 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.764227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:04.764426 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:04.765061 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:04.765101 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:04.768151 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":276,"threads_started":1}
W20260929 02:07:04.768544 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:04.759688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:42595 (local address 127.11.182.254:33325)
0929 02:07:04.759950 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:04.759960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:04.760307 (+ 347us) server_negotiation.cc:407] Connection header received
0929 02:07:04.760448 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:04.760452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:04.760511 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:04.760598 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:04.761420 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.762137 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.763488 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:04.764019 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:04.764155 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:04.768329 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:42595: BlockingRecv error: recv got EOF from 127.11.182.193:42595 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 02:07:05 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:05 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:05.231797 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:05.222815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:34355)
0929 02:07:05.223521 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:05.223536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:05.223719 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:05.224044 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:05.224049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:05.224518 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:05.224696 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:05.224705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.225693 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.225701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:05.226655 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:05.226667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.227431 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.227437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:05.227609 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:05.228351 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:05.228369 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:05.231399 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":414,"threads_started":1}
W20260929 02:07:05.231858 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:05.222896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:34355 (local address 127.11.182.254:33325)
0929 02:07:05.223206 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:05.223216 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:05.223625 (+ 409us) server_negotiation.cc:407] Connection header received
0929 02:07:05.223849 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:05.223859 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:05.223918 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:05.224020 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:05.224840 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.225551 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.226797 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.227275 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.227412 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:05.231614 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:34355: BlockingRecv error: recv got EOF from 127.11.182.194:34355 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 29 02:07:05 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:05 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:05.732746 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:05.723819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:33963)
0929 02:07:05.724434 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:05.724450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:05.724568 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:05.725033 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:05.725037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:05.725453 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:05.725617 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:05.725626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.726683 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.726687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:05.727695 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:05.727706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.728445 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.728450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:05.728580 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:05.729234 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:05.729253 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:05.732487 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":289,"threads_started":1}
W20260929 02:07:05.732820 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:05.723910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:33963 (local address 127.11.182.254:33325)
0929 02:07:05.724085 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:05.724090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:05.724533 (+ 443us) server_negotiation.cc:407] Connection header received
0929 02:07:05.724734 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:05.724738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:05.724824 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:05.724963 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:05.725785 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.726539 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.727829 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.728267 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.728368 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:05.732625 (+ 4257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:33963: BlockingRecv error: recv got EOF from 127.11.182.195:33963 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 02:07:05 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:05 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:05.777495 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:05.769342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:58515)
0929 02:07:05.769917 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:05.769932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:05.770075 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:05.770438 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:05.770443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:05.770903 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:05.771068 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:05.771077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.772065 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.772071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:05.773016 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:05.773027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.773778 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.773785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:05.773957 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:05.774522 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:05.774538 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:05.777271 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":265,"threads_started":1}
W20260929 02:07:05.777719 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:05.769543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:58515 (local address 127.11.182.254:33325)
0929 02:07:05.769774 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:05.769781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:05.770009 (+ 228us) server_negotiation.cc:407] Connection header received
0929 02:07:05.770231 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:05.770236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:05.770305 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:05.770421 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:05.771206 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.771921 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.773153 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:05.773595 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:05.773741 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:05.777448 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:58515: BlockingRecv error: recv got EOF from 127.11.182.193:58515 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 02:07:06 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:06 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:06.241595 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:06.233082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:48023)
0929 02:07:06.233614 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:06.233634 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:06.233807 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:06.234189 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:06.234193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:06.234790 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:06.235014 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:06.235024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.236204 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.236208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:06.237142 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:06.237154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.237839 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.237843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:06.237951 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:06.238621 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:06.238638 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:06.241372 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":243,"threads_started":1}
W20260929 02:07:06.241894 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:06.233177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:48023 (local address 127.11.182.254:33325)
0929 02:07:06.233425 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:06.233432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:06.233746 (+ 314us) server_negotiation.cc:407] Connection header received
0929 02:07:06.233965 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:06.233970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:06.234051 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:06.234174 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:06.235154 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.236072 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.237280 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.237679 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.237772 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:06.241584 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:48023: BlockingRecv error: recv got EOF from 127.11.182.194:48023 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 02:07:06 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:06 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:06.742681 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:06.733732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:35709)
0929 02:07:06.734265 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:06.734281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:06.734458 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:06.734918 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:06.734923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:06.735430 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:06.735618 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:06.735629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.736864 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.736868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:06.737849 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:06.737862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.738571 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.738576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:06.738691 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:06.739459 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:06.739478 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:06.742347 (+ 2869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
W20260929 02:07:06.742807 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:06.733841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:35709 (local address 127.11.182.254:33325)
0929 02:07:06.734058 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:06.734066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:06.734366 (+ 300us) server_negotiation.cc:407] Connection header received
0929 02:07:06.734604 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:06.734608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:06.734721 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:06.734834 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:06.735772 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.736724 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.737974 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.738364 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.738452 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:06.742568 (+ 4116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:35709: BlockingRecv error: recv got EOF from 127.11.182.195:35709 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:07:06 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:06 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:06.787312 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:06.778507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:46097)
0929 02:07:06.779197 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:06.779213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:06.779324 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:06.779643 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:06.779648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:06.780011 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:06.780182 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:06.780191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.781232 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.781236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:06.782171 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:06.782182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.783051 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.783056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:06.783211 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:06.783829 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:06.783846 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:06.786957 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":437,"threads_started":1}
W20260929 02:07:06.787374 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:06.778615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:46097 (local address 127.11.182.254:33325)
0929 02:07:06.778777 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:06.778781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:06.779346 (+ 565us) server_negotiation.cc:407] Connection header received
0929 02:07:06.779409 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:06.779412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:06.779504 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:06.779610 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:06.780341 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.781100 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.782339 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:06.782827 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:06.783018 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:06.787178 (+ 4160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:46097: BlockingRecv error: recv got EOF from 127.11.182.193:46097 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:07:07 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:07 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:07.252231 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:07.242781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:38087)
0929 02:07:07.243445 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:07.243461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:07.243632 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:07.243983 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:07.243988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:07.244392 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:07.244549 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:07.244558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.245674 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.245682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:07.246722 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:07.246734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.247496 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.247502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:07.247677 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:07.248471 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:07.248492 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:07.251839 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":409,"threads_started":1}
W20260929 02:07:07.252270 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:07.242893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:38087 (local address 127.11.182.254:33325)
0929 02:07:07.243143 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:07.243151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:07.243564 (+ 413us) server_negotiation.cc:407] Connection header received
0929 02:07:07.243773 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:07.243779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:07.243848 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:07.243954 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:07.244729 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.245546 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.246904 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.247331 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.247472 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:07.252056 (+ 4584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:38087: BlockingRecv error: recv got EOF from 127.11.182.194:38087 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 02:07:07 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:07 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:07.752444 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:07.743675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:57243)
0929 02:07:07.744226 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:07.744243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:07.744359 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:07.744765 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:07.744769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:07.745134 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:07.745303 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:07.745311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.746392 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.746397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:07.747444 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:07.747460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.748334 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.748341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:07.748527 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:07.749199 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:07.749225 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:07.752203 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":313,"threads_started":1}
W20260929 02:07:07.752797 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:07.743800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:57243 (local address 127.11.182.254:33325)
0929 02:07:07.743963 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:07.743968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:07.744345 (+ 377us) server_negotiation.cc:407] Connection header received
0929 02:07:07.744528 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:07.744536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:07.744632 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:07.744724 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:07.745467 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.746242 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.747604 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.748179 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.748277 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:07.752420 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:57243: BlockingRecv error: recv got EOF from 127.11.182.195:57243 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:07:07 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:07 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:07.796661 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:07.788417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:50265)
0929 02:07:07.788926 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:07.788941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:07.789130 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:07.789491 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:07.789495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:07.789896 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:07.790038 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:07.790046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.791128 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.791134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:07.792048 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:07.792058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.792814 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.792820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:07.792996 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:07.793628 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:07.793645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:07.796421 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
W20260929 02:07:07.796955 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:07.788481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:50265 (local address 127.11.182.254:33325)
0929 02:07:07.788742 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:07.788751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:07.789026 (+ 275us) server_negotiation.cc:407] Connection header received
0929 02:07:07.789281 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:07.789288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:07.789354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:07.789458 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:07.790187 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.790993 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.792222 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:07.792655 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:07.792784 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:07.796714 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:50265: BlockingRecv error: recv got EOF from 127.11.182.193:50265 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 02:07:08 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:08 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:08.262298 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:08.253386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:54493)
0929 02:07:08.253936 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:08.253957 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:08.254098 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:08.254432 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:08.254436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:08.254786 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:08.254948 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:08.254956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.256013 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.256017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:08.257108 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:08.257120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.257931 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.257936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:08.258134 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:08.258758 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:08.258778 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:08.262043 (+ 3265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":310,"threads_started":1}
W20260929 02:07:08.262552 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:08.253433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:54493 (local address 127.11.182.254:33325)
0929 02:07:08.253621 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:08.253630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:08.254047 (+ 417us) server_negotiation.cc:407] Connection header received
0929 02:07:08.254239 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:08.254243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:08.254305 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:08.254422 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:08.255124 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.255875 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.257261 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.257730 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.257837 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:08.262275 (+ 4438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:54493: BlockingRecv error: recv got EOF from 127.11.182.194:54493 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 02:07:08 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:08 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:08.761998 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:08.753564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:40579)
0929 02:07:08.754153 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:08.754170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:08.754323 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:08.754684 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:08.754689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:08.755202 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:08.755358 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:08.755366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.756429 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.756434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:08.757393 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:08.757403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.758106 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.758112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:08.758277 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:08.758891 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:08.758918 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:08.761644 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":322,"threads_started":1}
W20260929 02:07:08.762127 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:08.753856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:40579 (local address 127.11.182.254:33325)
0929 02:07:08.754075 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:08.754081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:08.754257 (+ 176us) server_negotiation.cc:407] Connection header received
0929 02:07:08.754449 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:08.754455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:08.754548 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:08.754653 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:08.755523 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.756295 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.757532 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.757949 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.758090 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:08.761833 (+ 3743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:40579: BlockingRecv error: recv got EOF from 127.11.182.195:40579 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 02:07:08 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:08 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:08.806200 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:08.797622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:34369)
0929 02:07:08.798190 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:08.798205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:08.798357 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:08.798691 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:08.798696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:08.799211 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:08.799387 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:08.799396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.800345 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.800353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:08.801251 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:08.801262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.801980 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.801986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:08.802166 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:08.802930 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:08.802953 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:08.805855 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":303,"threads_started":1}
W20260929 02:07:08.806345 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:08.797771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:34369 (local address 127.11.182.254:33325)
0929 02:07:08.798019 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:08.798026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:08.798289 (+ 263us) server_negotiation.cc:407] Connection header received
0929 02:07:08.798488 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:08.798497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:08.798561 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:08.798664 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:08.799533 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.800218 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.801391 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:08.801829 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:08.801964 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:08.806095 (+ 4131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:34369: BlockingRecv error: recv got EOF from 127.11.182.193:34369 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 02:07:09 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:09 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:09.272987 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:09.263447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:57279)
0929 02:07:09.264039 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:09.264084 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:09.264214 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:09.264568 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:09.264576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:09.265124 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:09.265339 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:09.265348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.266395 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.266399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:09.267501 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:09.267515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.268457 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.268462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:09.268593 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:09.269363 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:09.269391 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:09.272721 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":328,"threads_started":1}
W20260929 02:07:09.273269 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:09.263606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:57279 (local address 127.11.182.254:33325)
0929 02:07:09.263836 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:09.263842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:09.264198 (+ 356us) server_negotiation.cc:407] Connection header received
0929 02:07:09.264308 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:09.264314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:09.264400 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:09.264480 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:09.265488 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.266246 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.267661 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.268260 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.268374 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:09.272959 (+ 4585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:57279: BlockingRecv error: recv got EOF from 127.11.182.194:57279 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 02:07:09 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:09 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:09.779793 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:09.763191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:54883)
0929 02:07:09.763816 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:09.763841 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:09.764010 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:09.764472 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:09.764478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:09.765494 (+ 1016us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:09.765790 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:09.765805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.766992 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.766999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:09.768202 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:09.768217 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.768961 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.768967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:09.769147 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:09.776334 (+ 7187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:09.776357 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:09.779432 (+ 3075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":303,"threads_started":1}
W20260929 02:07:09.779919 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:09.763619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:54883 (local address 127.11.182.254:33325)
0929 02:07:09.763824 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:09.763830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:09.763936 (+ 106us) server_negotiation.cc:407] Connection header received
0929 02:07:09.764147 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:09.764152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:09.764303 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:09.764437 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:09.766005 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.766746 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.768353 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.768799 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.768937 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:09.779654 (+ 10717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:54883: BlockingRecv error: recv got EOF from 127.11.182.195:54883 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 02:07:09 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:09 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:09.815683 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:09.807221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:50823)
0929 02:07:09.807716 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:09.807731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:09.807883 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:09.808207 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:09.808213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:09.808600 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:09.808745 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:09.808753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.809717 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.809722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:09.810638 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:09.810648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.811440 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.811445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:09.811624 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:09.812259 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:09.812276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:09.815352 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":286,"threads_started":1}
W20260929 02:07:09.815824 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:09.807288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:50823 (local address 127.11.182.254:33325)
0929 02:07:09.807501 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:09.807509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:09.807809 (+ 300us) server_negotiation.cc:407] Connection header received
0929 02:07:09.808009 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:09.808013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:09.808070 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:09.808170 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:09.808878 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.809578 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.810774 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:09.811274 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:09.811412 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:09.815591 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:50823: BlockingRecv error: recv got EOF from 127.11.182.193:50823 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:07:10 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:10 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:10.283110 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:10.274099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:44245)
0929 02:07:10.274697 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:10.274721 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:10.274928 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:10.275304 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:10.275310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:10.275819 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:10.276025 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:10.276038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.277385 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.277389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:10.278369 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:10.278390 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.279246 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.279250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:10.279385 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:10.279995 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:10.280014 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:10.282811 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":323,"threads_started":1}
W20260929 02:07:10.283512 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:10.274280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:44245 (local address 127.11.182.254:33325)
0929 02:07:10.274554 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:10.274561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:10.274883 (+ 322us) server_negotiation.cc:407] Connection header received
0929 02:07:10.275070 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:10.275082 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:10.275147 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:10.275231 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:10.276199 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.277217 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.278541 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.279071 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.279195 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:10.283041 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:44245: BlockingRecv error: recv got EOF from 127.11.182.194:44245 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 02:07:10 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:10 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:10.789873 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:10.781130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:50863)
0929 02:07:10.781710 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:10.781729 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:10.781875 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:10.782260 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:10.782264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:10.782759 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:10.782998 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:10.783008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.783994 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.784000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:10.784912 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:10.784922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.785656 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.785661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:10.785835 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:10.786525 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:10.786548 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:10.789650 (+ 3102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":319,"threads_started":1}
W20260929 02:07:10.790163 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:10.781257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:50863 (local address 127.11.182.254:33325)
0929 02:07:10.781535 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:10.781542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:10.781834 (+ 292us) server_negotiation.cc:407] Connection header received
0929 02:07:10.782029 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:10.782034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:10.782116 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:10.782233 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:10.783132 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.783846 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.785063 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.785509 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.785644 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:10.789864 (+ 4220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:50863: BlockingRecv error: recv got EOF from 127.11.182.195:50863 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 02:07:10 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:10 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:10.825531 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:10.816876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:52755)
0929 02:07:10.817529 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:10.817547 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:10.817674 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:10.818062 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:10.818067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:10.818616 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:10.818780 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:10.818789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.819882 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.819889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:10.820846 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:10.820858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.821592 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.821598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:10.821768 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:10.822355 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:10.822372 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:10.825290 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":338,"threads_started":1}
W20260929 02:07:10.825753 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:10.817033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:52755 (local address 127.11.182.254:33325)
0929 02:07:10.817217 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:10.817221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:10.817640 (+ 419us) server_negotiation.cc:407] Connection header received
0929 02:07:10.817802 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:10.817807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:10.817896 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:10.818021 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:10.818982 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.819731 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.820993 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:10.821403 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:10.821536 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:10.825507 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:52755: BlockingRecv error: recv got EOF from 127.11.182.193:52755 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 02:07:11 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:11 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:11.292444 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:11.284042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:46755)
0929 02:07:11.284601 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:11.284617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:11.284731 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:11.285028 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:11.285033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:11.285443 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:11.285621 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:11.285630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.286721 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.286726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:11.287781 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:11.287795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.288563 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.288568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:11.288692 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:11.289343 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:11.289361 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:11.292194 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":307,"threads_started":1}
W20260929 02:07:11.292711 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:11.284264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:46755 (local address 127.11.182.254:33325)
0929 02:07:11.284452 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:11.284456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:11.284721 (+ 265us) server_negotiation.cc:407] Connection header received
0929 02:07:11.284829 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:11.284833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:11.284898 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:11.284976 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:11.285793 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.286538 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.287942 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.288391 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.288490 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:11.292427 (+ 3937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:46755: BlockingRecv error: recv got EOF from 127.11.182.194:46755 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 02:07:11 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:11 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:11.800086 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:11.790976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:33841)
0929 02:07:11.791563 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:11.791581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:11.791701 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:11.792172 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:11.792177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:11.792588 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:11.792816 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:11.792829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.793871 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.793875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:11.794815 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:11.794827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.796030 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.796037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:11.796211 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:11.796872 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:11.796891 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:11.799858 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":287,"threads_started":1}
W20260929 02:07:11.800294 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:11.791167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:33841 (local address 127.11.182.254:33325)
0929 02:07:11.791435 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:11.791441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:11.791669 (+ 228us) server_negotiation.cc:407] Connection header received
0929 02:07:11.791855 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:11.791859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:11.791993 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:11.792086 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:11.793001 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.793722 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.795112 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.795853 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.795991 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:11.800065 (+ 4074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:33841: BlockingRecv error: recv got EOF from 127.11.182.195:33841 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 02:07:11 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:11 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:11.836087 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:11.826657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:60161)
0929 02:07:11.827202 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:11.827221 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:11.827343 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:11.827675 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:11.827679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:11.828051 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:11.828248 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:11.828257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.829370 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.829375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:11.830460 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:11.830476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.831324 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.831330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:11.831523 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:11.832334 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:11.832358 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:11.835837 (+ 3479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":322,"threads_started":1}
W20260929 02:07:11.836447 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:11.826816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:60161 (local address 127.11.182.254:33325)
0929 02:07:11.827217 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:11.827223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:11.827305 (+ 82us) server_negotiation.cc:407] Connection header received
0929 02:07:11.827440 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:11.827443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:11.827522 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:11.827604 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:11.828405 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.829199 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.830651 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:11.831163 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:11.831270 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:11.836080 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:60161: BlockingRecv error: recv got EOF from 127.11.182.193:60161 (error 108)
Metrics: {"server-negotiator.queue_time_us":286}
Sep 29 02:07:12 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:12 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:12.301766 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:12.293562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:60339)
0929 02:07:12.294045 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:12.294062 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:12.294174 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:12.294458 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:12.294461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:12.294810 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:12.295068 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:12.295077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.296187 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.296191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:12.297075 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:12.297086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.297835 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.297838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:12.297957 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:12.298640 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:12.298658 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:12.301542 (+ 2884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":252,"threads_started":1}
W20260929 02:07:12.302089 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:12.293616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:60339 (local address 127.11.182.254:33325)
0929 02:07:12.293835 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:12.293845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:12.294165 (+ 320us) server_negotiation.cc:407] Connection header received
0929 02:07:12.294273 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:12.294277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:12.294336 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:12.294419 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:12.295221 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.296044 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.297221 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.297696 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.297823 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:12.301784 (+ 3961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:60339: BlockingRecv error: recv got EOF from 127.11.182.194:60339 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 02:07:12 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:12 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.195@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:12.809706 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:12.801076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.195:50787)
0929 02:07:12.801591 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:12.801628 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:12.801740 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:12.802069 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:12.802073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:12.802472 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:12.802669 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:12.802677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.803825 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.803829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:12.804735 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:12.804747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.805550 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.805555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:12.805735 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:12.806468 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:12.806487 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:12.809432 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":268,"threads_started":1}
W20260929 02:07:12.809841 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:12.801232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:50787 (local address 127.11.182.254:33325)
0929 02:07:12.801406 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:12.801413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:12.801710 (+ 297us) server_negotiation.cc:407] Connection header received
0929 02:07:12.801842 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:12.801846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:12.801916 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:12.802043 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:12.802827 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.803680 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.804877 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.805361 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.805525 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:12.809643 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.195:50787: BlockingRecv error: recv got EOF from 127.11.182.195:50787 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 02:07:12 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:12 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.193@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:12.846074 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:12.837331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.193:33887)
0929 02:07:12.837857 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:12.837874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:12.838036 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:12.838379 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:12.838384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:12.838823 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:12.839054 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:12.839063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.840031 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.840036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:12.840990 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:12.841001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.841717 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.841722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:12.841904 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:12.842550 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:12.842567 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:12.845695 (+ 3128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":306,"threads_started":1}
W20260929 02:07:12.846237 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:12.837383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:33887 (local address 127.11.182.254:33325)
0929 02:07:12.837636 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:12.837646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:12.837965 (+ 319us) server_negotiation.cc:407] Connection header received
0929 02:07:12.838169 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:12.838174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:12.838240 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:12.838353 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:12.839202 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.839887 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.841140 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:12.841560 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:12.841690 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:12.845979 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.193:33887: BlockingRecv error: recv got EOF from 127.11.182.193:33887 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 29 02:07:13 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:07:13 dist-test-slave-q06p krb5kdc[20236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.182.194@KRBTEST.COM for oryx/127.11.182.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:07:13.311980 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:13.303024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33325 (local address 127.11.182.194:45909)
0929 02:07:13.303570 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:13.303587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:13.303751 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:13.304124 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:13.304128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:13.304531 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:13.304689 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:13.304698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.305757 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.305764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:13.306691 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:13.306703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.307474 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.307480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:13.307646 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:13.308345 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:13.308361 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:13.311633 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.182.254:33325: Server oryx/127.11.182.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":281,"threads_started":1}
W20260929 02:07:13.312036 21240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:13.303080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:45909 (local address 127.11.182.254:33325)
0929 02:07:13.303335 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:13.303345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:13.303679 (+ 334us) server_negotiation.cc:407] Connection header received
0929 02:07:13.303878 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:13.303883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:13.303954 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:13.304061 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:13.304835 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.305613 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.306835 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.307313 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.307456 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:13.311818 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.182.194:45909: BlockingRecv error: recv got EOF from 127.11.182.194:45909 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:07:13.621755 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 20825
I20260929 02:07:13.634814 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:13.622933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.11.182.193:34339)
0929 02:07:13.623500 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:13.623507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:13.623535 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:07:13.623596 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:13.623602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:13.623819 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:13.623981 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:13.625666 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.626720 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.628113 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.628720 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.628833 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:13.631487 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:13.631496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:13.631510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:13.631544 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:13.634254 (+ 2710us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.182.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:07:13.634521 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54030: Request ticket server kudu/127.11.182.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":274,"threads_started":1}
W20260929 02:07:13.634977 21286 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54030: Request ticket server kudu/127.11.182.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:07:13.635107 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 20825: Not authorized: Client connection negotiation failed: client connection to 127.11.182.193:34339: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.182.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:07:13.635322 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 20825
I20260929 02:07:13.652276 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 20977
I20260929 02:07:13.665405 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:13.653378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.11.182.194:46619)
0929 02:07:13.654016 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:13.654028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:13.654069 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:07:13.654145 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:13.654157 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:13.654479 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:13.654698 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:13.656298 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.657452 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.658832 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.659559 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.659664 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:13.662251 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:13.662261 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:13.662275 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:13.662309 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:13.664876 (+ 2567us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.182.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:07:13.665136 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46748: Request ticket server kudu/127.11.182.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":283,"threads_started":1}
W20260929 02:07:13.665509 21287 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46748: Request ticket server kudu/127.11.182.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:07:13.665627 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 20977: Not authorized: Client connection negotiation failed: client connection to 127.11.182.194:46619: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.182.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:07:13.665791 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 20977
I20260929 02:07:13.682338 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 21108
I20260929 02:07:13.694205 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:13.683384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.11.182.195:46857)
0929 02:07:13.683950 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:13.683958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:13.683983 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:07:13.684041 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:13.684047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:13.684282 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:13.684426 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:13.685746 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.686787 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.688091 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:13.688648 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:13.688751 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:13.691332 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:13.691341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:13.691357 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:13.691398 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:13.693704 (+ 2306us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.182.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:07:13.693951 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45318: Request ticket server kudu/127.11.182.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":279,"threads_started":1}
W20260929 02:07:13.694330 21288 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45318: Request ticket server kudu/127.11.182.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:07:13.694350 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 21108: Not authorized: Client connection negotiation failed: client connection to 127.11.182.195:46857: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.182.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:07:13.694556 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 21108
I20260929 02:07:13.710631 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 20756
I20260929 02:07:13.711884 20818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:13.824079 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 20756
2026-09-29T02:07:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21346 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T02:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:07:13Z Disabled control of system clock
I20260929 02:07:13.900672 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:42725
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:42725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:07:14.232012 21293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:14.232324 21293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:14.232404 21293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:14.239679 21293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:07:14.239797 21293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:14.239835 21293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:14.239887 21293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:07:14.239933 21293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:07:14.249086 21293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:42725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:42725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:14.250347 21293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:14.252025 21293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:14.264117 21298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:14.264143 21301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:14.264746 21299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:14.264923 21293 server_base.cc:1034] running on GCE node
I20260929 02:07:14.265781 21293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:14.267308 21293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:14.268568 21293 hybrid_clock.cc:648] HybridClock initialized: now 1790647634268499 us; error 67 us; skew 500 ppm
I20260929 02:07:14.268931 21293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:07:14.271401 21293 webserver.cc:460] Webserver started at http://127.11.182.254:35947/ using document root <none> and password file <none>
I20260929 02:07:14.272112 21293 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:14.272243 21293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:14.272578 21293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:14.275550 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "893e1a7eae684a61b6c7922c87efef90"
format_stamp: "Formatted at 2026-09-29 02:07:14 on dist-test-slave-q06p"
server_key: "8279970672a6517dbad1a66cb39ba714"
server_key_iv: "15f0e97163e1eabcdcef56ade19ed3d8"
server_key_version: "encryptionkey@0"
I20260929 02:07:14.276321 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "893e1a7eae684a61b6c7922c87efef90"
format_stamp: "Formatted at 2026-09-29 02:07:14 on dist-test-slave-q06p"
server_key: "8279970672a6517dbad1a66cb39ba714"
server_key_iv: "15f0e97163e1eabcdcef56ade19ed3d8"
server_key_version: "encryptionkey@0"
I20260929 02:07:14.281441 21293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:07:14.285104 21307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:14.286507 21293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 02:07:14.286695 21293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "893e1a7eae684a61b6c7922c87efef90"
format_stamp: "Formatted at 2026-09-29 02:07:14 on dist-test-slave-q06p"
server_key: "8279970672a6517dbad1a66cb39ba714"
server_key_iv: "15f0e97163e1eabcdcef56ade19ed3d8"
server_key_version: "encryptionkey@0"
I20260929 02:07:14.286962 21293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:14.299966 21293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:14.300812 21293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:14.301100 21293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:14.319773 21293 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:42725
I20260929 02:07:14.319782 21358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:42725 every 8 connection(s)
I20260929 02:07:14.321089 21293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:07:14.322567 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 21293
I20260929 02:07:14.322760 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:07:14.323427 11995 external_mini_cluster.cc:1428] Setting key a853bd2c588c7b5790fb8c4699b18d3e
I20260929 02:07:14.325325 21359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:07:14.336403 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:14.325394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.11.182.254:42725)
0929 02:07:14.326349 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:14.326359 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:14.326399 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:07:14.327996 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:14.328017 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:14.328457 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:14.328852 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:14.329423 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:14.330714 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:14.332055 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:14.332701 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:14.332886 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:14.334075 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:14.334083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:14.334099 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:14.334679 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:14.334751 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:14.334770 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:14.334922 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:14.335168 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 02:07:14.335458 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":327,"threads_started":1}
I20260929 02:07:14.337116 21359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90: Bootstrap starting.
I20260929 02:07:14.340255 21359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90: Neither blocks nor log segments found. Creating new log.
I20260929 02:07:14.341224 21359 log.cc:824] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90: Log is configured to *not* fsync() on all Append() calls
I20260929 02:07:14.343894 21359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90: No bootstrap required, opened a new log
I20260929 02:07:14.351372 21359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893e1a7eae684a61b6c7922c87efef90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 42725 } }
I20260929 02:07:14.351802 21359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:07:14.351913 21359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893e1a7eae684a61b6c7922c87efef90, State: Initialized, Role: FOLLOWER
I20260929 02:07:14.352407 21359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893e1a7eae684a61b6c7922c87efef90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 42725 } }
I20260929 02:07:14.352593 21359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:07:14.352687 21359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:07:14.352806 21359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:07:14.354818 21359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893e1a7eae684a61b6c7922c87efef90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 42725 } }
I20260929 02:07:14.355321 21359 leader_election.cc:302] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 893e1a7eae684a61b6c7922c87efef90; no voters:
I20260929 02:07:14.356055 21359 leader_election.cc:288] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:07:14.356174 21364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:07:14.356936 21364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [term 1 LEADER]: Becoming Leader. State: Replica: 893e1a7eae684a61b6c7922c87efef90, State: Running, Role: LEADER
I20260929 02:07:14.357461 21364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893e1a7eae684a61b6c7922c87efef90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 42725 } }
I20260929 02:07:14.358264 21359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:07:14.360903 21366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 893e1a7eae684a61b6c7922c87efef90. Latest consensus state: current_term: 1 leader_uuid: "893e1a7eae684a61b6c7922c87efef90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893e1a7eae684a61b6c7922c87efef90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 42725 } } }
I20260929 02:07:14.361366 21366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [sys.catalog]: This master's current role is: LEADER
I20260929 02:07:14.362918 21374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:07:14.362641 21365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "893e1a7eae684a61b6c7922c87efef90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893e1a7eae684a61b6c7922c87efef90" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 42725 } } }
I20260929 02:07:14.363181 21365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90 [sys.catalog]: This master's current role is: LEADER
I20260929 02:07:14.367586 21374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:07:14.376367 21374 catalog_manager.cc:1347] Generated new cluster ID: 9aa8816c6ef745f78a8ba7b253e53df6
I20260929 02:07:14.376497 21374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:07:14.401235 21374 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:07:14.403036 21374 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:07:14.411944 21374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 893e1a7eae684a61b6c7922c87efef90: Generated new TSK 0
I20260929 02:07:14.412966 21374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:07:14.425724 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:42725
--builtin_ntp_servers=127.11.182.212:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 02:07:14.762960 21383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:14.763288 21383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:14.763446 21383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:14.771013 21383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:14.771154 21383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:14.771332 21383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:07:14.780411 21383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:42725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:14.781697 21383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:14.783527 21383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:14.797066 21388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:14.797062 21391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:14.797045 21389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:14.798169 21383 server_base.cc:1034] running on GCE node
I20260929 02:07:14.798811 21383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:14.799794 21383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:14.801092 21383 hybrid_clock.cc:648] HybridClock initialized: now 1790647634801039 us; error 44 us; skew 500 ppm
I20260929 02:07:14.801424 21383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:07:14.814426 21383 webserver.cc:460] Webserver started at http://127.11.182.193:43919/ using document root <none> and password file <none>
I20260929 02:07:14.815330 21383 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:14.815474 21383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:14.815820 21383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:14.818957 21383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "abf919ff624b473b8498c5ffed2a2d18"
format_stamp: "Formatted at 2026-09-29 02:07:14 on dist-test-slave-q06p"
server_key: "9ed2e70089dae3cefb4f0af1872138cd"
server_key_iv: "bc16f664f2e4e0d8d9ab1ee842eef176"
server_key_version: "encryptionkey@0"
I20260929 02:07:14.819814 21383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "abf919ff624b473b8498c5ffed2a2d18"
format_stamp: "Formatted at 2026-09-29 02:07:14 on dist-test-slave-q06p"
server_key: "9ed2e70089dae3cefb4f0af1872138cd"
server_key_iv: "bc16f664f2e4e0d8d9ab1ee842eef176"
server_key_version: "encryptionkey@0"
I20260929 02:07:14.825176 21383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:07:14.828758 21397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:14.830122 21383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:07:14.830323 21383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "abf919ff624b473b8498c5ffed2a2d18"
format_stamp: "Formatted at 2026-09-29 02:07:14 on dist-test-slave-q06p"
server_key: "9ed2e70089dae3cefb4f0af1872138cd"
server_key_iv: "bc16f664f2e4e0d8d9ab1ee842eef176"
server_key_version: "encryptionkey@0"
I20260929 02:07:14.830554 21383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:14.846016 21383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:14.846939 21383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:14.847235 21383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:14.848152 21383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:07:14.849555 21383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:07:14.849675 21383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:14.849813 21383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:07:14.849902 21383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:14.884400 21383 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:40957
I20260929 02:07:14.884498 21510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:40957 every 8 connection(s)
I20260929 02:07:14.885869 21383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:07:14.887984 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 21383
I20260929 02:07:14.888195 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:07:14.889035 11995 external_mini_cluster.cc:1428] Setting key b4f8cd2aa3f0c9e4d16520dbad0b12e7
I20260929 02:07:14.892159 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:42725
--builtin_ntp_servers=127.11.182.212:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:07:14.900115 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:14.888876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:35425 (local address 127.11.182.254:42725)
0929 02:07:14.889377 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:14.889382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:14.891161 (+ 1779us) server_negotiation.cc:407] Connection header received
0929 02:07:14.892841 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:14.892851 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:14.892944 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:14.893066 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:14.894178 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:14.895318 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:14.896942 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:14.897665 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:14.897804 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:14.899267 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:14.899273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:14.899278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:14.899308 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:14.899344 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:14.899356 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:14.899462 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:14.899680 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 02:07:14.899775 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":285,"threads_started":1}
I20260929 02:07:14.901165 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:14.889599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:42725 (local address 127.11.182.193:35425)
0929 02:07:14.890771 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:14.890824 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:14.892568 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:14.893284 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:14.893292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:14.893384 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:14.893997 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:14.894024 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:14.895490 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:14.895494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:14.896791 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:14.896803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:14.897871 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:14.897876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:14.898093 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:14.898923 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:07:14.898926 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:14.898973 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:07:14.899055 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:07:14.899460 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:07:14.899472 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:07:14.899482 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:07:14.899673 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 02:07:14.900088 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":342,"threads_started":1}
I20260929 02:07:14.902735 21511 heartbeater.cc:347] Connected to a master server at 127.11.182.254:42725
I20260929 02:07:14.903064 21511 heartbeater.cc:464] Registering TS with master...
I20260929 02:07:14.903825 21511 heartbeater.cc:511] Master 127.11.182.254:42725 requested a full tablet report, sending...
I20260929 02:07:14.905866 21324 ts_manager.cc:194] Registered new tserver with Master: abf919ff624b473b8498c5ffed2a2d18 (127.11.182.193:40957)
I20260929 02:07:14.907402 21324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.193:35425
W20260929 02:07:15.222954 21515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:15.223295 21515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:15.223451 21515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:15.231007 21515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:15.231173 21515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:15.231338 21515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:07:15.240849 21515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:42725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:15.242228 21515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:15.244040 21515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:15.257395 21523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:15.257444 21520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:15.257493 21521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:15.260942 21515 server_base.cc:1034] running on GCE node
I20260929 02:07:15.261646 21515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:15.262879 21515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:15.264113 21515 hybrid_clock.cc:648] HybridClock initialized: now 1790647635264046 us; error 69 us; skew 500 ppm
I20260929 02:07:15.264420 21515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:07:15.266906 21515 webserver.cc:460] Webserver started at http://127.11.182.194:39207/ using document root <none> and password file <none>
I20260929 02:07:15.267596 21515 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:15.267685 21515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:15.267956 21515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:15.271039 21515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "83443f8f428b4f86a397271f9aa79edf"
format_stamp: "Formatted at 2026-09-29 02:07:15 on dist-test-slave-q06p"
server_key: "83f95ee1f85caf416f415ac7176021be"
server_key_iv: "38b84b92ae7aa9792c6f3060743b912c"
server_key_version: "encryptionkey@0"
I20260929 02:07:15.271795 21515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "83443f8f428b4f86a397271f9aa79edf"
format_stamp: "Formatted at 2026-09-29 02:07:15 on dist-test-slave-q06p"
server_key: "83f95ee1f85caf416f415ac7176021be"
server_key_iv: "38b84b92ae7aa9792c6f3060743b912c"
server_key_version: "encryptionkey@0"
I20260929 02:07:15.277141 21515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 02:07:15.280751 21529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:15.282289 21515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:07:15.282442 21515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "83443f8f428b4f86a397271f9aa79edf"
format_stamp: "Formatted at 2026-09-29 02:07:15 on dist-test-slave-q06p"
server_key: "83f95ee1f85caf416f415ac7176021be"
server_key_iv: "38b84b92ae7aa9792c6f3060743b912c"
server_key_version: "encryptionkey@0"
I20260929 02:07:15.282662 21515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:15.301028 21515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:15.301961 21515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:15.302282 21515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:15.303263 21515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:07:15.304708 21515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:07:15.304834 21515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:15.304947 21515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:07:15.305028 21515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:15.343461 21515 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:37067
I20260929 02:07:15.343510 21641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:37067 every 8 connection(s)
I20260929 02:07:15.344739 21515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:07:15.346236 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 21515
I20260929 02:07:15.346388 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:07:15.346894 11995 external_mini_cluster.cc:1428] Setting key a9d374cbd276856b456b70ed3d4a0b94
I20260929 02:07:15.349743 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:42725
--builtin_ntp_servers=127.11.182.212:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:07:15.357628 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:15.347354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:54993 (local address 127.11.182.254:42725)
0929 02:07:15.347569 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:15.347576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:15.349339 (+ 1763us) server_negotiation.cc:407] Connection header received
0929 02:07:15.351251 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:15.351256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:15.351338 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:15.351462 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:15.352504 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.353220 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.354762 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.355244 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.355381 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:15.356738 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:15.356747 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:15.356755 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:15.356780 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:15.356821 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:15.356838 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:15.356934 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:15.357128 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 02:07:15.357234 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:07:15.358465 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:15.347997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:42725 (local address 127.11.182.194:54993)
0929 02:07:15.349202 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:15.349237 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:15.350952 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:15.351690 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:15.351698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:15.351787 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:15.352342 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:15.352359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.353371 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.353378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:15.354616 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:15.354627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.355466 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.355473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:15.355701 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:15.356388 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:07:15.356391 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:15.356429 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:07:15.356503 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:07:15.356938 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:07:15.356946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:07:15.356951 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:07:15.357092 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 02:07:15.357374 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":532,"threads_started":1}
I20260929 02:07:15.359984 21642 heartbeater.cc:347] Connected to a master server at 127.11.182.254:42725
I20260929 02:07:15.360293 21642 heartbeater.cc:464] Registering TS with master...
I20260929 02:07:15.361034 21642 heartbeater.cc:511] Master 127.11.182.254:42725 requested a full tablet report, sending...
I20260929 02:07:15.362438 21324 ts_manager.cc:194] Registered new tserver with Master: 83443f8f428b4f86a397271f9aa79edf (127.11.182.194:37067)
I20260929 02:07:15.363313 21324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.194:54993
W20260929 02:07:15.684885 21645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:15.685170 21645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:15.685316 21645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:15.693001 21645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:15.693111 21645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:15.693287 21645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:07:15.702677 21645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:42725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:15.704028 21645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:15.705766 21645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:15.718750 21653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:15.718734 21650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:15.718787 21651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:15.719456 21645 server_base.cc:1034] running on GCE node
I20260929 02:07:15.720098 21645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:15.720971 21645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:15.722177 21645 hybrid_clock.cc:648] HybridClock initialized: now 1790647635722137 us; error 32 us; skew 500 ppm
I20260929 02:07:15.722508 21645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:07:15.724768 21645 webserver.cc:460] Webserver started at http://127.11.182.195:44387/ using document root <none> and password file <none>
I20260929 02:07:15.725467 21645 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:15.725555 21645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:15.725901 21645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:15.728775 21645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "e82b3d04ef65434f907b8fba73ac02bc"
format_stamp: "Formatted at 2026-09-29 02:07:15 on dist-test-slave-q06p"
server_key: "dd4ada72136d081073a28c2b08b6d087"
server_key_iv: "c666cdb6ed28c5985acb3c8c65cf8449"
server_key_version: "encryptionkey@0"
I20260929 02:07:15.729591 21645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "e82b3d04ef65434f907b8fba73ac02bc"
format_stamp: "Formatted at 2026-09-29 02:07:15 on dist-test-slave-q06p"
server_key: "dd4ada72136d081073a28c2b08b6d087"
server_key_iv: "c666cdb6ed28c5985acb3c8c65cf8449"
server_key_version: "encryptionkey@0"
I20260929 02:07:15.734735 21645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:07:15.738325 21659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:15.739740 21645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:07:15.739921 21645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "e82b3d04ef65434f907b8fba73ac02bc"
format_stamp: "Formatted at 2026-09-29 02:07:15 on dist-test-slave-q06p"
server_key: "dd4ada72136d081073a28c2b08b6d087"
server_key_iv: "c666cdb6ed28c5985acb3c8c65cf8449"
server_key_version: "encryptionkey@0"
I20260929 02:07:15.740145 21645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:15.757356 21645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:15.758262 21645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:15.758555 21645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:15.759651 21645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:07:15.761116 21645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:07:15.761226 21645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:15.761358 21645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:07:15.761442 21645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:15.794909 21645 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:39963
I20260929 02:07:15.795024 21771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:39963 every 8 connection(s)
I20260929 02:07:15.796257 21645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:07:15.803373 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 21645
I20260929 02:07:15.803548 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:07:15.804093 11995 external_mini_cluster.cc:1428] Setting key f760f0583947223a5988a601229cfaad
I20260929 02:07:15.808584 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:15.798666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:34085 (local address 127.11.182.254:42725)
0929 02:07:15.798892 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:15.798899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:15.800281 (+ 1382us) server_negotiation.cc:407] Connection header received
0929 02:07:15.801921 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:15.801925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:15.801990 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:15.802079 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:15.803152 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.804277 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.806122 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.806556 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.806652 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:15.807934 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:15.807939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:15.807942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:15.807972 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:15.808002 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:15.808013 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:15.808082 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:15.808276 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 02:07:15.808347 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:07:15.809568 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:15.799299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:42725 (local address 127.11.182.195:34085)
0929 02:07:15.800139 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:15.800173 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:15.801677 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:15.802265 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:15.802272 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:15.802351 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:15.802984 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:15.803005 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.804418 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.804422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:15.805999 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:15.806012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.806771 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.806779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:15.807047 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:15.807665 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:07:15.807667 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:15.807703 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:07:15.807772 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:07:15.808095 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:07:15.808102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:07:15.808108 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:07:15.808234 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 02:07:15.808505 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":262,"threads_started":1}
I20260929 02:07:15.810806 21772 heartbeater.cc:347] Connected to a master server at 127.11.182.254:42725
I20260929 02:07:15.811131 21772 heartbeater.cc:464] Registering TS with master...
I20260929 02:07:15.811847 21772 heartbeater.cc:511] Master 127.11.182.254:42725 requested a full tablet report, sending...
I20260929 02:07:15.813170 21324 ts_manager.cc:194] Registered new tserver with Master: e82b3d04ef65434f907b8fba73ac02bc (127.11.182.195:39963)
I20260929 02:07:15.813953 21324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.195:34085
I20260929 02:07:15.819348 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:07:15.825227 21513 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:15.823617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.11.182.254:42725)
0929 02:07:15.823885 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:15.823895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:15.824226 (+ 331us) server_negotiation.cc:407] Connection header received
0929 02:07:15.824333 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:15.824337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:15.824411 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:15.824501 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:15.824980 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43754: BlockingRecv error: recv got EOF from 127.0.0.1:43754 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:07:15.826072 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 21383
I20260929 02:07:15.833477 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:15.827038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.11.182.193:40957)
0929 02:07:15.827551 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:15.827558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:15.827582 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:07:15.827657 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:15.827663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:15.827869 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:15.827987 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:15.828350 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.829331 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.830503 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.831119 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.831213 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:15.832128 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:15.832136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:15.832149 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:15.832734 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:15.832776 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:15.832794 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:15.832884 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:15.832996 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 02:07:15.833097 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":298,"threads_started":1}
I20260929 02:07:15.834735 21506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:15.914300 21511 heartbeater.cc:503] Master 127.11.182.254:42725 was elected leader, sending a full tablet report...
I20260929 02:07:15.915645 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 21383
W20260929 02:07:15.932101 21311 connection.cc:535] server connection from 127.11.182.193:35425 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:07:15.932247 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 21515
I20260929 02:07:15.940071 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:15.933165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.11.182.194:37067)
0929 02:07:15.933709 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:15.933716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:15.933742 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:07:15.933804 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:15.933811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:15.934024 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:15.934151 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:15.934548 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.935591 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.936832 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:15.937455 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:15.937566 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:15.938616 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:15.938626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:15.938643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:15.939326 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:15.939382 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:15.939402 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:15.939490 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:15.939654 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 02:07:15.939776 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":292,"threads_started":1}
I20260929 02:07:15.941344 21637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:16.023350 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 21515
I20260929 02:07:16.039638 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 21645
I20260929 02:07:16.048502 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:16.040729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.11.182.195:39963)
0929 02:07:16.041409 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:16.041416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:16.041448 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:07:16.041540 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:16.041547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:16.041755 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:16.041882 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:16.042331 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:16.043522 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:16.044996 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:16.045585 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:16.045684 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:16.046950 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:16.046961 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:16.046976 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:16.047573 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:16.047652 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:16.047673 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:16.047819 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:16.047994 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 02:07:16.048142 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":347,"threads_started":1}
I20260929 02:07:16.049873 21767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:16.133263 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 21645
I20260929 02:07:16.149132 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 21293
I20260929 02:07:16.150287 21354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:16.250131 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 21293
2026-09-29T02:07:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2421 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T02:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:07:16Z Disabled control of system clock
I20260929 02:07:16.321399 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45061
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:07:16.650717 21791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:16.651139 21791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:16.651233 21791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:16.658471 21791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:07:16.658588 21791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:16.658627 21791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:16.658675 21791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:07:16.658720 21791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:07:16.669417 21791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:45061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:16.670670 21791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:16.672508 21791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:16.684535 21796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:16.684454 21797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:16.684506 21799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:16.685364 21791 server_base.cc:1034] running on GCE node
I20260929 02:07:16.686164 21791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:16.687742 21791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:16.689033 21791 hybrid_clock.cc:648] HybridClock initialized: now 1790647636688958 us; error 55 us; skew 500 ppm
I20260929 02:07:16.689450 21791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:07:16.692010 21791 webserver.cc:460] Webserver started at http://127.11.182.254:36695/ using document root <none> and password file <none>
I20260929 02:07:16.692714 21791 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:16.692845 21791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:16.693171 21791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:16.696300 21791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "c57d5989172b48c1b49d943ab35f37bc"
format_stamp: "Formatted at 2026-09-29 02:07:16 on dist-test-slave-q06p"
server_key: "06f0b867b36a8ba258a98651d1d8ac02"
server_key_iv: "e2ac6ae177277d2f391b8022c542d6bf"
server_key_version: "encryptionkey@0"
I20260929 02:07:16.697115 21791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "c57d5989172b48c1b49d943ab35f37bc"
format_stamp: "Formatted at 2026-09-29 02:07:16 on dist-test-slave-q06p"
server_key: "06f0b867b36a8ba258a98651d1d8ac02"
server_key_iv: "e2ac6ae177277d2f391b8022c542d6bf"
server_key_version: "encryptionkey@0"
I20260929 02:07:16.702267 21791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:07:16.705709 21805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:16.707048 21791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:07:16.707213 21791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "c57d5989172b48c1b49d943ab35f37bc"
format_stamp: "Formatted at 2026-09-29 02:07:16 on dist-test-slave-q06p"
server_key: "06f0b867b36a8ba258a98651d1d8ac02"
server_key_iv: "e2ac6ae177277d2f391b8022c542d6bf"
server_key_version: "encryptionkey@0"
I20260929 02:07:16.707423 21791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:16.712843 21791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:16.713572 21791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:16.713866 21791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:16.731567 21791 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:45061
I20260929 02:07:16.731599 21856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:45061 every 8 connection(s)
I20260929 02:07:16.732944 21791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:07:16.736526 21857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:07:16.743002 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 21791
I20260929 02:07:16.743223 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:07:16.743863 11995 external_mini_cluster.cc:1428] Setting key 2cda924d9940a1887283ac7bfbf28628
I20260929 02:07:16.746997 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc: Bootstrap starting.
I20260929 02:07:16.751021 21857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:07:16.752403 21857 log.cc:824] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc: Log is configured to *not* fsync() on all Append() calls
I20260929 02:07:16.753309 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:16.746000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.11.182.254:45061)
0929 02:07:16.747066 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:16.747081 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:16.747134 (+ 53us) server_negotiation.cc:407] Connection header received
0929 02:07:16.749470 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:16.749498 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:16.750005 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:16.750412 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:16.751460 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:16.751468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:16.751487 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:16.751511 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:16.751600 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:16.751633 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:16.751800 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:16.751926 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 02:07:16.752181 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":304,"threads_started":1}
I20260929 02:07:16.755191 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc: No bootstrap required, opened a new log
I20260929 02:07:16.763034 21857 raft_consensus.cc:348] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57d5989172b48c1b49d943ab35f37bc" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45061 } }
I20260929 02:07:16.763478 21857 raft_consensus.cc:374] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:07:16.763597 21857 raft_consensus.cc:729] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57d5989172b48c1b49d943ab35f37bc, State: Initialized, Role: FOLLOWER
I20260929 02:07:16.764362 21857 consensus_queue.cc:260] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57d5989172b48c1b49d943ab35f37bc" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45061 } }
I20260929 02:07:16.764614 21857 raft_consensus.cc:388] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:07:16.764797 21857 raft_consensus.cc:482] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:07:16.764981 21857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:07:16.767349 21857 raft_consensus.cc:504] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57d5989172b48c1b49d943ab35f37bc" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45061 } }
I20260929 02:07:16.767947 21857 leader_election.cc:302] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c57d5989172b48c1b49d943ab35f37bc; no voters:
I20260929 02:07:16.768574 21857 leader_election.cc:288] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:07:16.768746 21862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:07:16.769507 21862 raft_consensus.cc:686] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [term 1 LEADER]: Becoming Leader. State: Replica: c57d5989172b48c1b49d943ab35f37bc, State: Running, Role: LEADER
I20260929 02:07:16.770165 21862 consensus_queue.cc:237] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57d5989172b48c1b49d943ab35f37bc" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45061 } }
I20260929 02:07:16.771008 21857 sys_catalog.cc:564] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:07:16.773361 21864 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c57d5989172b48c1b49d943ab35f37bc. Latest consensus state: current_term: 1 leader_uuid: "c57d5989172b48c1b49d943ab35f37bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57d5989172b48c1b49d943ab35f37bc" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45061 } } }
I20260929 02:07:16.773887 21864 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [sys.catalog]: This master's current role is: LEADER
I20260929 02:07:16.773769 21863 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c57d5989172b48c1b49d943ab35f37bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57d5989172b48c1b49d943ab35f37bc" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45061 } } }
I20260929 02:07:16.774087 21863 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc [sys.catalog]: This master's current role is: LEADER
I20260929 02:07:16.775810 21870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:07:16.781399 21870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:07:16.790256 21870 catalog_manager.cc:1347] Generated new cluster ID: 9f3b17e0625f4c29b5cbdca514125aa5
I20260929 02:07:16.790364 21870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:07:16.840252 21870 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:07:16.840785 21870 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:07:16.849252 21870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c57d5989172b48c1b49d943ab35f37bc: Generated new TSK 0
I20260929 02:07:16.850095 21870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:07:16.862195 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45061
--builtin_ntp_servers=127.11.182.212:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:07:17.198284 21881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:17.198580 21881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:17.198670 21881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:17.206002 21881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:17.206100 21881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:17.206211 21881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:07:17.215394 21881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:17.216629 21881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:17.218416 21881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:17.231393 21886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:17.231449 21887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:17.231360 21889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:17.232111 21881 server_base.cc:1034] running on GCE node
I20260929 02:07:17.232681 21881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:17.233633 21881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:17.234911 21881 hybrid_clock.cc:648] HybridClock initialized: now 1790647637234813 us; error 45 us; skew 500 ppm
I20260929 02:07:17.235222 21881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:07:17.237571 21881 webserver.cc:460] Webserver started at http://127.11.182.193:36283/ using document root <none> and password file <none>
I20260929 02:07:17.238242 21881 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:17.238329 21881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:17.238590 21881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:17.241588 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "bdd3f98b0d6f47ffa1c303a78cddc37f"
format_stamp: "Formatted at 2026-09-29 02:07:17 on dist-test-slave-q06p"
server_key: "7dfcd47ce4c892cb05b6a192c4602eee"
server_key_iv: "d0aed2df43bc4b87cd0e5b86bec604a0"
server_key_version: "encryptionkey@0"
I20260929 02:07:17.242319 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "bdd3f98b0d6f47ffa1c303a78cddc37f"
format_stamp: "Formatted at 2026-09-29 02:07:17 on dist-test-slave-q06p"
server_key: "7dfcd47ce4c892cb05b6a192c4602eee"
server_key_iv: "d0aed2df43bc4b87cd0e5b86bec604a0"
server_key_version: "encryptionkey@0"
I20260929 02:07:17.247381 21881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 02:07:17.250648 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:17.252004 21881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:07:17.252159 21881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "bdd3f98b0d6f47ffa1c303a78cddc37f"
format_stamp: "Formatted at 2026-09-29 02:07:17 on dist-test-slave-q06p"
server_key: "7dfcd47ce4c892cb05b6a192c4602eee"
server_key_iv: "d0aed2df43bc4b87cd0e5b86bec604a0"
server_key_version: "encryptionkey@0"
I20260929 02:07:17.252373 21881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:17.258090 21881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:17.258939 21881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:17.259217 21881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:17.260113 21881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:07:17.261485 21881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:07:17.261571 21881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:17.261698 21881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:07:17.261798 21881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:17.295140 21881 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:43815
I20260929 02:07:17.295212 22007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:43815 every 8 connection(s)
I20260929 02:07:17.296519 21881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:07:17.304749 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:17.299273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:43905 (local address 127.11.182.254:45061)
0929 02:07:17.299715 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:17.299720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:17.300866 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 02:07:17.302507 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:17.302512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:17.302583 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:17.302707 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:17.303992 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:17.303998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:17.304003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:17.304021 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:17.304069 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:17.304080 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:17.304204 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:17.304382 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 02:07:17.304447 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260929 02:07:17.305315 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 21881
I20260929 02:07:17.305531 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:07:17.305727 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:17.299839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45061 (local address 127.11.182.193:43905)
0929 02:07:17.300720 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:17.300755 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:17.302248 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:17.302975 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:17.302983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:17.303075 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:17.303720 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:07:17.303722 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:17.303757 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:07:17.303837 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:07:17.304176 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:07:17.304186 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:07:17.304193 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:07:17.304467 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 02:07:17.304724 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":265,"threads_started":1}
I20260929 02:07:17.306331 11995 external_mini_cluster.cc:1428] Setting key 57d6fe56cee2b8e12f9c8bb8ee4a04c4
I20260929 02:07:17.307266 22008 heartbeater.cc:347] Connected to a master server at 127.11.182.254:45061
I20260929 02:07:17.307543 22008 heartbeater.cc:464] Registering TS with master...
I20260929 02:07:17.308163 22008 heartbeater.cc:511] Master 127.11.182.254:45061 requested a full tablet report, sending...
I20260929 02:07:17.309369 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45061
--builtin_ntp_servers=127.11.182.212:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 02:07:17.310379 21822 ts_manager.cc:194] Registered new tserver with Master: bdd3f98b0d6f47ffa1c303a78cddc37f (127.11.182.193:43815)
W20260929 02:07:17.633450 22012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:17.633752 22012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:17.633922 22012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:17.641185 22012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:17.641281 22012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:17.641460 22012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:07:17.650450 22012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:17.651789 22012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:17.653503 22012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:17.667001 22017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:17.666963 22018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:17.667044 22020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:17.668229 22012 server_base.cc:1034] running on GCE node
I20260929 02:07:17.669032 22012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:17.670332 22012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:17.671720 22012 hybrid_clock.cc:648] HybridClock initialized: now 1790647637671610 us; error 123 us; skew 500 ppm
I20260929 02:07:17.672102 22012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:07:17.674348 22012 webserver.cc:460] Webserver started at http://127.11.182.194:38167/ using document root <none> and password file <none>
I20260929 02:07:17.675143 22012 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:17.675230 22012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:17.675589 22012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:17.678416 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "6a59b495d5a74b4da2cd814029335142"
format_stamp: "Formatted at 2026-09-29 02:07:17 on dist-test-slave-q06p"
server_key: "c1914d9986a527303438c8b5fdab71b5"
server_key_iv: "3329ecab8b261a5b6077e6dcf05e33f4"
server_key_version: "encryptionkey@0"
I20260929 02:07:17.679248 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "6a59b495d5a74b4da2cd814029335142"
format_stamp: "Formatted at 2026-09-29 02:07:17 on dist-test-slave-q06p"
server_key: "c1914d9986a527303438c8b5fdab71b5"
server_key_iv: "3329ecab8b261a5b6077e6dcf05e33f4"
server_key_version: "encryptionkey@0"
I20260929 02:07:17.684293 22012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 02:07:17.687649 22026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:17.688964 22012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:07:17.689123 22012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "6a59b495d5a74b4da2cd814029335142"
format_stamp: "Formatted at 2026-09-29 02:07:17 on dist-test-slave-q06p"
server_key: "c1914d9986a527303438c8b5fdab71b5"
server_key_iv: "3329ecab8b261a5b6077e6dcf05e33f4"
server_key_version: "encryptionkey@0"
I20260929 02:07:17.689352 22012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:17.694991 22012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:17.695735 22012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:17.695994 22012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:17.696918 22012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:07:17.698271 22012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:07:17.698375 22012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:17.698523 22012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:07:17.698596 22012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:17.732321 22012 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:40619
I20260929 02:07:17.732364 22138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:40619 every 8 connection(s)
I20260929 02:07:17.733632 22012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:07:17.741966 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:17.736288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:60211 (local address 127.11.182.254:45061)
0929 02:07:17.736456 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:17.736460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:17.737976 (+ 1516us) server_negotiation.cc:407] Connection header received
0929 02:07:17.739697 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:17.739702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:17.739791 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:17.739906 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:17.741180 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:17.741186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:17.741191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:17.741210 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:17.741248 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:17.741259 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:17.741404 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:17.741591 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 02:07:17.741663 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:07:17.742159 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 22012
I20260929 02:07:17.742337 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:07:17.742873 11995 external_mini_cluster.cc:1428] Setting key ebbb67b3ac8f0d1a1e12e29fd7815b9f
I20260929 02:07:17.743075 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:17.736907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45061 (local address 127.11.182.194:60211)
0929 02:07:17.737826 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:17.737865 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:17.739440 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:17.740129 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:17.740138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:17.740228 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:17.740902 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:07:17.740905 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:17.740940 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:07:17.741019 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:07:17.741369 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:07:17.741381 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:07:17.741390 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:07:17.741670 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 02:07:17.741918 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":283,"threads_started":1}
I20260929 02:07:17.744566 22139 heartbeater.cc:347] Connected to a master server at 127.11.182.254:45061
I20260929 02:07:17.744879 22139 heartbeater.cc:464] Registering TS with master...
I20260929 02:07:17.745579 22139 heartbeater.cc:511] Master 127.11.182.254:45061 requested a full tablet report, sending...
I20260929 02:07:17.745990 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45061
--builtin_ntp_servers=127.11.182.212:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 02:07:17.747118 21821 ts_manager.cc:194] Registered new tserver with Master: 6a59b495d5a74b4da2cd814029335142 (127.11.182.194:40619)
W20260929 02:07:18.111931 22142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:18.112267 22142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:18.112442 22142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:18.120154 22142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:18.120337 22142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:18.120548 22142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:07:18.130628 22142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:18.132030 22142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:18.133885 22142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:18.148627 22148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:18.148728 22147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:18.149194 22142 server_base.cc:1034] running on GCE node
W20260929 02:07:18.148728 22150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:18.150051 22142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:18.151249 22142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:18.152509 22142 hybrid_clock.cc:648] HybridClock initialized: now 1790647638152433 us; error 57 us; skew 500 ppm
I20260929 02:07:18.152961 22142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:07:18.155612 22142 webserver.cc:460] Webserver started at http://127.11.182.195:34369/ using document root <none> and password file <none>
I20260929 02:07:18.156497 22142 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:18.156622 22142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:18.156986 22142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:18.160205 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "aaa449e999af422da832e9392f8c2c5f"
format_stamp: "Formatted at 2026-09-29 02:07:18 on dist-test-slave-q06p"
server_key: "161b8b1876c3e37725e87c46248c755d"
server_key_iv: "d9fbb07d345c2dbebcdcd2768310a33a"
server_key_version: "encryptionkey@0"
I20260929 02:07:18.161039 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "aaa449e999af422da832e9392f8c2c5f"
format_stamp: "Formatted at 2026-09-29 02:07:18 on dist-test-slave-q06p"
server_key: "161b8b1876c3e37725e87c46248c755d"
server_key_iv: "d9fbb07d345c2dbebcdcd2768310a33a"
server_key_version: "encryptionkey@0"
I20260929 02:07:18.166687 22142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:07:18.170583 22156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:18.172338 22142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:07:18.172528 22142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "aaa449e999af422da832e9392f8c2c5f"
format_stamp: "Formatted at 2026-09-29 02:07:18 on dist-test-slave-q06p"
server_key: "161b8b1876c3e37725e87c46248c755d"
server_key_iv: "d9fbb07d345c2dbebcdcd2768310a33a"
server_key_version: "encryptionkey@0"
I20260929 02:07:18.172758 22142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:18.179020 22142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:18.179903 22142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:18.180188 22142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:18.181167 22142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:07:18.182587 22142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:07:18.182713 22142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:18.182875 22142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:07:18.182960 22142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 02:07:18.217725 22142 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:32789
I20260929 02:07:18.217775 22268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:32789 every 8 connection(s)
I20260929 02:07:18.219305 22142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:07:18.220412 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 22142
I20260929 02:07:18.220614 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:07:18.221184 11995 external_mini_cluster.cc:1428] Setting key 3c31a1325ce9c95d0fc2566c0ea65f77
I20260929 02:07:18.228863 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:18.222372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:41005 (local address 127.11.182.254:45061)
0929 02:07:18.222622 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:18.222628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:18.224597 (+ 1969us) server_negotiation.cc:407] Connection header received
0929 02:07:18.226342 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:18.226347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:18.226409 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:18.226510 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:18.228183 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:18.228188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:18.228191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:18.228209 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:18.228242 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:18.228254 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:18.228373 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:18.228568 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 02:07:18.228627 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:07:18.229827 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:18.223148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45061 (local address 127.11.182.195:41005)
0929 02:07:18.224374 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:18.224451 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:18.226023 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:18.226795 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:18.226819 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:18.227020 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:18.227822 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:07:18.227825 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:18.227869 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:07:18.227983 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:07:18.228352 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:07:18.228359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:07:18.228365 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:07:18.228615 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 02:07:18.228840 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":384,"threads_started":1}
I20260929 02:07:18.231225 22269 heartbeater.cc:347] Connected to a master server at 127.11.182.254:45061
I20260929 02:07:18.231536 22269 heartbeater.cc:464] Registering TS with master...
I20260929 02:07:18.232137 22269 heartbeater.cc:511] Master 127.11.182.254:45061 requested a full tablet report, sending...
I20260929 02:07:18.233659 21821 ts_manager.cc:194] Registered new tserver with Master: aaa449e999af422da832e9392f8c2c5f (127.11.182.195:32789)
I20260929 02:07:18.237419 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:07:18.243562 22010 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:07:18.241951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.11.182.254:45061)
0929 02:07:18.242122 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:18.242126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:18.242645 (+ 519us) server_negotiation.cc:407] Connection header received
0929 02:07:18.242779 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:18.242783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:18.242832 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:18.242972 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:18.243285 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47528: BlockingRecv error: recv got EOF from 127.0.0.1:47528 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:07:18.244346 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 21881
I20260929 02:07:18.248422 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:18.245343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.11.182.193:43815)
0929 02:07:18.245906 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:18.245913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:18.245944 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:07:18.246015 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:18.246021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:18.246247 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:18.246385 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:18.247824 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:18.247831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:18.247843 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:18.247865 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:18.247911 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:18.247927 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:18.248040 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:18.248112 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 02:07:18.248192 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":296,"threads_started":1}
I20260929 02:07:18.249676 22003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:18.330725 22008 heartbeater.cc:503] Master 127.11.182.254:45061 was elected leader, sending a full tablet report...
I20260929 02:07:18.331923 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 21881
W20260929 02:07:18.349474 21806 connection.cc:535] server connection from 127.11.182.193:43905 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:07:18.349923 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 22012
I20260929 02:07:18.353451 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:18.350920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.11.182.194:40619)
0929 02:07:18.351477 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:18.351484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:18.351511 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:07:18.351572 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:18.351577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:18.351805 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:18.351937 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:18.352824 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:18.352830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:18.352840 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:18.352858 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:18.352896 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:18.352910 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:18.353013 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:18.353146 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 02:07:18.353222 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":285,"threads_started":1}
I20260929 02:07:18.354657 22134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:18.436159 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 22012
I20260929 02:07:18.452581 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 22142
I20260929 02:07:18.456498 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:18.453638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.11.182.195:32789)
0929 02:07:18.454319 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:18.454326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:18.454366 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:07:18.454426 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:18.454433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:18.454668 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:18.454798 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:18.455772 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:18.455778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:18.455789 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:07:18.455805 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:18.455843 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:07:18.455856 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:18.455986 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:18.456039 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 02:07:18.456152 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":353,"threads_started":1}
I20260929 02:07:18.457767 22264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:18.540158 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 22142
I20260929 02:07:18.556572 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 21791
I20260929 02:07:18.557765 21852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:18.659300 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 21791
2026-09-29T02:07:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2409 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:07:18 dist-test-slave-q06p krb5kdc[22287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:07:18 dist-test-slave-q06p krb5kdc[22287](info): set up 2 sockets
Sep 29 02:07:18 dist-test-slave-q06p krb5kdc[22287](info): commencing operation
krb5kdc: starting...
W20260929 02:07:19.768548 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:07:19 dist-test-slave-q06p krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:07:20.008458 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40849
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:07:20.338696 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:20.339054 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:20.339238 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:20.346324 22300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:07:20.346477 22300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:20.346593 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:20.346697 22300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:07:20.346765 22300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:07:20.356557 22300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:40849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:20.357890 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:20.359205 22300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:20.370723 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:20.375083 22308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:20.370811 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:20.376365 22300 server_base.cc:1034] running on GCE node
I20260929 02:07:20.377241 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:20.378784 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:20.380077 22300 hybrid_clock.cc:648] HybridClock initialized: now 1790647640380003 us; error 65 us; skew 500 ppm
I20260929 02:07:20.380470 22300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:07:20 dist-test-slave-q06p krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647640, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:07:20.389058 22300 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:07:20.390511 22300 webserver.cc:460] Webserver started at http://127.11.182.254:43857/ using document root <none> and password file <none>
I20260929 02:07:20.391242 22300 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:20.391345 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:20.391584 22300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:20.394428 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "dcef0865afc64ab88d89ba470e331f56"
format_stamp: "Formatted at 2026-09-29 02:07:20 on dist-test-slave-q06p"
server_key: "f389c09fce639a238c7d74a8f658b571"
server_key_iv: "680b5a663190eaf27b77bae0221f5c61"
server_key_version: "encryptionkey@0"
I20260929 02:07:20.395203 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "dcef0865afc64ab88d89ba470e331f56"
format_stamp: "Formatted at 2026-09-29 02:07:20 on dist-test-slave-q06p"
server_key: "f389c09fce639a238c7d74a8f658b571"
server_key_iv: "680b5a663190eaf27b77bae0221f5c61"
server_key_version: "encryptionkey@0"
I20260929 02:07:20.400401 22300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:07:20.403745 22315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:20.405011 22300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:07:20.405201 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "dcef0865afc64ab88d89ba470e331f56"
format_stamp: "Formatted at 2026-09-29 02:07:20 on dist-test-slave-q06p"
server_key: "f389c09fce639a238c7d74a8f658b571"
server_key_iv: "680b5a663190eaf27b77bae0221f5c61"
server_key_version: "encryptionkey@0"
I20260929 02:07:20.405367 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:20.439993 22300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:20.444891 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:20.445210 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:20.464521 22300 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:40849
I20260929 02:07:20.464615 22366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:40849 every 8 connection(s)
I20260929 02:07:20.465976 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:07:20.469792 22367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:07:20.471683 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 22300
I20260929 02:07:20.471913 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:07:20.472702 11995 external_mini_cluster.cc:1428] Setting key d9a3eab5e449b009a6575e82dc729f5b
I20260929 02:07:20.482589 22367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56: Bootstrap starting.
I20260929 02:07:20.486635 22367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56: Neither blocks nor log segments found. Creating new log.
Sep 29 02:07:20 dist-test-slave-q06p krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:07:20.487787 22367 log.cc:824] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56: Log is configured to *not* fsync() on all Append() calls
I20260929 02:07:20.491243 22367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56: No bootstrap required, opened a new log
I20260929 02:07:20.496905 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:20.474925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.11.182.254:40849)
0929 02:07:20.475768 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:20.475780 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:20.475824 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:07:20.477449 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:20.477473 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:20.477845 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:20.478163 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:20.479969 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:20.481278 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:20.483056 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:20.483722 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:20.483861 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:20.488505 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:20.488517 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:20.488532 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:20.488571 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:20.492250 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:20.493006 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:20.493020 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:20.493029 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:20.493235 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:20.493621 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:20.493626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:20.493630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:20.494386 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:20.494711 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:20.495062 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 02:07:20.495504 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":395,"threads_started":1}
I20260929 02:07:20.501698 22367 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcef0865afc64ab88d89ba470e331f56" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40849 } }
I20260929 02:07:20.502058 22367 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:07:20.502130 22367 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcef0865afc64ab88d89ba470e331f56, State: Initialized, Role: FOLLOWER
I20260929 02:07:20.502676 22367 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcef0865afc64ab88d89ba470e331f56" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40849 } }
I20260929 02:07:20.502954 22367 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:07:20.503062 22367 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:07:20.503173 22367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:07:20.505254 22367 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcef0865afc64ab88d89ba470e331f56" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40849 } }
I20260929 02:07:20.505692 22367 leader_election.cc:302] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcef0865afc64ab88d89ba470e331f56; no voters:
I20260929 02:07:20.506313 22367 leader_election.cc:288] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:07:20.506420 22372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:07:20.507375 22372 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [term 1 LEADER]: Becoming Leader. State: Replica: dcef0865afc64ab88d89ba470e331f56, State: Running, Role: LEADER
I20260929 02:07:20.507905 22372 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcef0865afc64ab88d89ba470e331f56" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40849 } }
I20260929 02:07:20.508489 22367 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:07:20.510694 22373 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcef0865afc64ab88d89ba470e331f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcef0865afc64ab88d89ba470e331f56" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40849 } } }
I20260929 02:07:20.511092 22373 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [sys.catalog]: This master's current role is: LEADER
I20260929 02:07:20.510878 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcef0865afc64ab88d89ba470e331f56. Latest consensus state: current_term: 1 leader_uuid: "dcef0865afc64ab88d89ba470e331f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcef0865afc64ab88d89ba470e331f56" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40849 } } }
I20260929 02:07:20.511303 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56 [sys.catalog]: This master's current role is: LEADER
I20260929 02:07:20.514019 22378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:07:20.519596 22378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:07:20.529515 22378 catalog_manager.cc:1347] Generated new cluster ID: 6f350923283f4ad0967fb2c6f3632e29
I20260929 02:07:20.529636 22378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:07:20.543607 22378 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:07:20.545423 22378 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:07:20.556982 22378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56: Generated new TSK 0
I20260929 02:07:20.558043 22378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:07:20.623291 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40849
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:07:20.957055 22393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:20.957357 22393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:20.957522 22393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:20.964967 22393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:20.965144 22393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:20.965317 22393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:07:20.975082 22393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:20.976435 22393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:20.977708 22393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:20.990710 22401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:20.990829 22398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:20.990729 22399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:20.996407 22393 server_base.cc:1034] running on GCE node
I20260929 02:07:20.997180 22393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:20.998176 22393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:20.999406 22393 hybrid_clock.cc:648] HybridClock initialized: now 1790647640999361 us; error 47 us; skew 500 ppm
I20260929 02:07:20.999773 22393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:07:20 dist-test-slave-q06p krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647640, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:07:21.002936 22393 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:07:21.004417 22393 webserver.cc:460] Webserver started at http://127.11.182.193:37963/ using document root <none> and password file <none>
I20260929 02:07:21.005132 22393 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:21.005278 22393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:21.005584 22393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:21.008543 22393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "890aaebb0f4c48a3868073d3f0d72faa"
format_stamp: "Formatted at 2026-09-29 02:07:20 on dist-test-slave-q06p"
server_key: "b4280e63a02b2042ffc6bd83bd2070f7"
server_key_iv: "ed2d6dab5071ec39c132511fa9d6b718"
server_key_version: "encryptionkey@0"
I20260929 02:07:21.009305 22393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "890aaebb0f4c48a3868073d3f0d72faa"
format_stamp: "Formatted at 2026-09-29 02:07:20 on dist-test-slave-q06p"
server_key: "b4280e63a02b2042ffc6bd83bd2070f7"
server_key_iv: "ed2d6dab5071ec39c132511fa9d6b718"
server_key_version: "encryptionkey@0"
I20260929 02:07:21.014518 22393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:07:21.018031 22408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:21.019550 22393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:07:21.019737 22393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "890aaebb0f4c48a3868073d3f0d72faa"
format_stamp: "Formatted at 2026-09-29 02:07:20 on dist-test-slave-q06p"
server_key: "b4280e63a02b2042ffc6bd83bd2070f7"
server_key_iv: "ed2d6dab5071ec39c132511fa9d6b718"
server_key_version: "encryptionkey@0"
I20260929 02:07:21.019951 22393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:21.034126 22393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:21.038072 22393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:21.038360 22393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:21.039368 22393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:07:21.040688 22393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:07:21.040789 22393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:21.040932 22393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:07:21.041006 22393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:21.074047 22393 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:37629
I20260929 02:07:21.074082 22520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:37629 every 8 connection(s)
I20260929 02:07:21.075419 22393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:07:21.078392 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 22393
I20260929 02:07:21.078636 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:07:21.079499 11995 external_mini_cluster.cc:1428] Setting key 9e0224498a010a68d5ec97a9970a5add
Sep 29 02:07:21 dist-test-slave-q06p krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647640, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:07:21.095745 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:21.077837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:41113 (local address 127.11.182.254:40849)
0929 02:07:21.078291 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:21.078295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:21.079438 (+ 1143us) server_negotiation.cc:407] Connection header received
0929 02:07:21.081418 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:21.081423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:21.081501 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:21.081619 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:21.083068 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:21.083767 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:21.085325 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:21.085725 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:21.085858 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:21.089722 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:21.089727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:21.089730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:21.089762 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:21.092350 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:21.093296 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:21.093300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:21.093302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:21.093415 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:21.093935 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:21.093939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:21.093941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:21.094199 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:21.094314 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:21.095298 (+ 984us) server_negotiation.cc:299] Negotiation successful
0929 02:07:21.095465 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":263,"threads_started":1}
I20260929 02:07:21.097308 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:21.078466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40849 (local address 127.11.182.193:41113)
0929 02:07:21.079290 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:21.079327 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:21.081183 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:21.081846 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:21.081854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:21.082316 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:21.082918 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:21.082937 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:21.083930 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:21.083940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:21.085170 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:21.085182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:21.085947 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:21.085956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:21.086194 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:21.086924 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:21.086949 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:21.089481 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:07:21.092507 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:07:21.092519 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:07:21.092541 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:07:21.093174 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:07:21.093530 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:07:21.093537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:07:21.093544 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:07:21.093813 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:07:21.094328 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:07:21.094336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:07:21.094840 (+ 504us) client_negotiation.cc:769] Sending connection context
0929 02:07:21.095206 (+ 366us) client_negotiation.cc:240] Negotiation successful
0929 02:07:21.095806 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":351,"threads_started":1}
I20260929 02:07:21.098879 22521 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40849
I20260929 02:07:21.099220 22521 heartbeater.cc:464] Registering TS with master...
I20260929 02:07:21.099969 22521 heartbeater.cc:511] Master 127.11.182.254:40849 requested a full tablet report, sending...
I20260929 02:07:21.102005 22332 ts_manager.cc:194] Registered new tserver with Master: 890aaebb0f4c48a3868073d3f0d72faa (127.11.182.193:37629)
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:07:21.103616 22332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:41113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:07:21.133971 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40849
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:07:21.456158 22527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:21.456498 22527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:21.456662 22527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:21.464573 22527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:21.464764 22527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:21.464969 22527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:07:21.475703 22527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:21.476969 22527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:21.478327 22527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:21.492182 22532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:21.492225 22535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:21.492166 22533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:21.492908 22527 server_base.cc:1034] running on GCE node
I20260929 02:07:21.493530 22527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:21.494480 22527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:21.495757 22527 hybrid_clock.cc:648] HybridClock initialized: now 1790647641495713 us; error 55 us; skew 500 ppm
I20260929 02:07:21.496129 22527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:07:21 dist-test-slave-q06p krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647641, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:07:21.499455 22527 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:07:21.501173 22527 webserver.cc:460] Webserver started at http://127.11.182.194:35287/ using document root <none> and password file <none>
I20260929 02:07:21.501860 22527 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:21.501984 22527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:21.502291 22527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:21.505455 22527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "b56599c2e473441e83d6632e57214f3c"
format_stamp: "Formatted at 2026-09-29 02:07:21 on dist-test-slave-q06p"
server_key: "7110a088bbca1cccf58e00c8f94361df"
server_key_iv: "dc9d09f425816b05823505b72b15ba47"
server_key_version: "encryptionkey@0"
I20260929 02:07:21.506230 22527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "b56599c2e473441e83d6632e57214f3c"
format_stamp: "Formatted at 2026-09-29 02:07:21 on dist-test-slave-q06p"
server_key: "7110a088bbca1cccf58e00c8f94361df"
server_key_iv: "dc9d09f425816b05823505b72b15ba47"
server_key_version: "encryptionkey@0"
I20260929 02:07:21.511751 22527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:07:21.515061 22542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:21.516363 22527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:07:21.516530 22527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "b56599c2e473441e83d6632e57214f3c"
format_stamp: "Formatted at 2026-09-29 02:07:21 on dist-test-slave-q06p"
server_key: "7110a088bbca1cccf58e00c8f94361df"
server_key_iv: "dc9d09f425816b05823505b72b15ba47"
server_key_version: "encryptionkey@0"
I20260929 02:07:21.516738 22527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:21.530702 22527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:21.534330 22527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:21.534622 22527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:21.535621 22527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:07:21.536892 22527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:07:21.536970 22527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:21.537142 22527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:07:21.537199 22527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:21.581646 22527 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:40237
I20260929 02:07:21.581651 22654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:40237 every 8 connection(s)
I20260929 02:07:21.583439 22527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:07:21.585830 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 22527
I20260929 02:07:21.586043 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:07:21.586480 11995 external_mini_cluster.cc:1428] Setting key 5b3a8aa291e036e6dfa42ae2d3694bf5
Sep 29 02:07:21 dist-test-slave-q06p krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647641, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:07:21.611446 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:21.587070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:42149 (local address 127.11.182.254:40849)
0929 02:07:21.587288 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:21.587295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:21.588672 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 02:07:21.591261 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:21.591270 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:21.591360 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:21.591485 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:21.593439 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:21.594694 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:21.597224 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:21.597953 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:21.598412 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:21.604824 (+ 6412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:21.604831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:21.604835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:21.604876 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:21.608356 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:21.609099 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:21.609105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:21.609109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:21.609244 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:21.609737 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:21.609743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:21.609746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:21.610018 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:21.610168 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:21.610884 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 02:07:21.611111 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:07:21.612382 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:21.587497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40849 (local address 127.11.182.194:42149)
0929 02:07:21.588394 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:21.588456 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:21.590962 (+ 2506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:21.591724 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:21.591737 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:21.592468 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:21.593244 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:21.593271 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:21.594885 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:21.594894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:21.597037 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:21.597054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:21.598726 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:21.598736 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:21.599147 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:21.599975 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:21.600011 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:21.604543 (+ 4532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:07:21.608496 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:07:21.608505 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:07:21.608523 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:07:21.608992 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:07:21.609370 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:07:21.609375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:07:21.609379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:07:21.609616 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:07:21.610143 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:07:21.610152 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:07:21.610501 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 02:07:21.610761 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 02:07:21.611308 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":350,"threads_started":1}
I20260929 02:07:21.613790 22655 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40849
I20260929 02:07:21.614105 22655 heartbeater.cc:464] Registering TS with master...
I20260929 02:07:21.614974 22655 heartbeater.cc:511] Master 127.11.182.254:40849 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
I20260929 02:07:21.616685 22332 ts_manager.cc:194] Registered new tserver with Master: b56599c2e473441e83d6632e57214f3c (127.11.182.194:40237)
I20260929 02:07:21.617507 22332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:42149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:07:21.641664 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40849
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:07:21.959631 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:07:21.959935 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:07:21.960106 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:07:21.967167 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:07:21.967314 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:07:21.967484 22660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:07:21.976862 22660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:40849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:07:21.978152 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:07:21.979523 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:07:21.992376 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:21.992319 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:07:21.992331 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:07:21.993230 22660 server_base.cc:1034] running on GCE node
I20260929 02:07:21.993814 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:07:21.994670 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:07:21.995941 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790647641995887 us; error 62 us; skew 500 ppm
I20260929 02:07:21.996310 22660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:07:21 dist-test-slave-q06p krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647641, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:07:21.999446 22660 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:07:22.000818 22660 webserver.cc:460] Webserver started at http://127.11.182.195:42547/ using document root <none> and password file <none>
I20260929 02:07:22.001538 22660 fs_manager.cc:359] Metadata directory not provided
I20260929 02:07:22.001662 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:07:22.001968 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:07:22.004899 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "2c9bf4158fe3462698c284242064b599"
format_stamp: "Formatted at 2026-09-29 02:07:21 on dist-test-slave-q06p"
server_key: "c67814ff4f4153fe9ec45299adae28fe"
server_key_iv: "79862814da0c69f465775ad0a3004616"
server_key_version: "encryptionkey@0"
I20260929 02:07:22.005659 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "2c9bf4158fe3462698c284242064b599"
format_stamp: "Formatted at 2026-09-29 02:07:21 on dist-test-slave-q06p"
server_key: "c67814ff4f4153fe9ec45299adae28fe"
server_key_iv: "79862814da0c69f465775ad0a3004616"
server_key_version: "encryptionkey@0"
I20260929 02:07:22.011070 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:07:22.014302 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:22.015851 22660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:07:22.016028 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "2c9bf4158fe3462698c284242064b599"
format_stamp: "Formatted at 2026-09-29 02:07:21 on dist-test-slave-q06p"
server_key: "c67814ff4f4153fe9ec45299adae28fe"
server_key_iv: "79862814da0c69f465775ad0a3004616"
server_key_version: "encryptionkey@0"
I20260929 02:07:22.016222 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:07:22.037305 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:07:22.041697 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:07:22.041993 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:07:22.043000 22660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:07:22.044302 22660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:07:22.044377 22660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:22.044549 22660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:07:22.044600 22660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:07:22.076550 22660 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:34689
I20260929 02:07:22.076604 22787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:34689 every 8 connection(s)
I20260929 02:07:22.077724 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:07:22.082357 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 22660
I20260929 02:07:22.082561 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:07:22.083014 11995 external_mini_cluster.cc:1428] Setting key ec523ed5656b79d4b4ee78b3878402d4
Sep 29 02:07:22 dist-test-slave-q06p krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647641, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:07:22.096800 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.080352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:51961 (local address 127.11.182.254:40849)
0929 02:07:22.080525 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.080529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.081602 (+ 1073us) server_negotiation.cc:407] Connection header received
0929 02:07:22.083607 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.083613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.083723 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.083828 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:22.085260 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.086112 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.087691 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.088079 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.088210 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.091441 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:22.091451 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:22.091454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:22.091485 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:22.093837 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.094569 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.094579 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.094582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.094702 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.095155 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.095160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.095163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.095423 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:22.095576 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.096248 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.096456 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:07:22.097945 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.080791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:40849 (local address 127.11.182.195:51961)
0929 02:07:22.081434 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:22.081493 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:22.083345 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:22.084074 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:22.084082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:22.084577 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:07:22.085103 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.085119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.086259 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.086266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.087548 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.087560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.088296 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.088304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.088536 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.089124 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:07:22.089149 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:07:22.091247 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:07:22.093995 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:07:22.094004 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:07:22.094020 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:07:22.094432 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:07:22.094813 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:07:22.094818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:07:22.094822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:07:22.095037 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:07:22.095548 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:07:22.095556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:07:22.095894 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 02:07:22.096155 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 02:07:22.096684 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":251,"threads_started":1}
I20260929 02:07:22.099375 22788 heartbeater.cc:347] Connected to a master server at 127.11.182.254:40849
I20260929 02:07:22.099682 22788 heartbeater.cc:464] Registering TS with master...
I20260929 02:07:22.100379 22788 heartbeater.cc:511] Master 127.11.182.254:40849 requested a full tablet report, sending...
I20260929 02:07:22.101671 22332 ts_manager.cc:194] Registered new tserver with Master: 2c9bf4158fe3462698c284242064b599 (127.11.182.195:34689)
I20260929 02:07:22.102451 22332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:51961
I20260929 02:07:22.107035 22521 heartbeater.cc:503] Master 127.11.182.254:40849 was elected leader, sending a full tablet report...
I20260929 02:07:22.110170 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:07:22.127038 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.114629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.11.182.254:40849)
0929 02:07:22.114785 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.114790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.115294 (+ 504us) server_negotiation.cc:407] Connection header received
0929 02:07:22.115420 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.115424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.115487 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.115568 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:22.116872 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.117588 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.118705 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.119206 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.119339 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.121773 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:22.121790 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:22.121794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:22.121831 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:22.124669 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.125283 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.125288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.125290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.125463 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.125804 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.125808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.125811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.126142 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:22.126326 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.126576 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.126728 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:07:22.130976 22332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:07:22.137619 22332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:07:22.174226 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.148984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:40237 (local address 127.0.0.1:37048)
0929 02:07:22.161811 (+ 12827us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:22.161862 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:22.162071 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:22.163503 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:22.163514 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:22.163548 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:07:22.164377 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.164394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.167416 (+ 3022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.167426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.171008 (+ 3582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.171026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.172976 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.172986 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.173262 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.173337 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 02:07:22.173478 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 02:07:22.173692 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12496,"thread_start_us":20372,"threads_started":1}
I20260929 02:07:22.175009 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.149116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:37629 (local address 127.0.0.1:35092)
0929 02:07:22.163201 (+ 14085us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:22.163225 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:22.163382 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:22.165038 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:22.165046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:22.165082 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:07:22.165630 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.165646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.168196 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.168204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.171899 (+ 3695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.171916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.174112 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.174118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.174282 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.174443 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 02:07:22.174534 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:07:22.174661 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13942,"thread_start_us":12210,"threads_started":1}
I20260929 02:07:22.175257 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.149837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.11.182.193:37629)
0929 02:07:22.162760 (+ 12923us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.162772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.163995 (+ 1223us) server_negotiation.cc:407] Connection header received
0929 02:07:22.164269 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.164280 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.164556 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.164755 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:07:22.165882 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.167996 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.172079 (+ 4083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.173582 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.173770 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.173956 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.174626 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.174838 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12743,"thread_start_us":12593,"threads_started":1}
I20260929 02:07:22.175801 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.149085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.11.182.194:40237)
0929 02:07:22.162708 (+ 13623us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.162722 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.162759 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:07:22.162907 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.162918 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.163241 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.163509 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:07:22.164684 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.167165 (+ 2481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.171193 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.172650 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.174190 (+ 1540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.174954 (+ 764us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.175044 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.175442 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13404,"thread_start_us":14660,"threads_started":1}
I20260929 02:07:22.177585 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.149455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:34689 (local address 127.0.0.1:33342)
0929 02:07:22.169682 (+ 20227us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:22.169706 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:22.169841 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:22.170759 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:22.170765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:22.170788 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:07:22.171250 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.171259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.172905 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.172923 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.175108 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.175127 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.176863 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.176870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.177040 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.177075 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:07:22.177160 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 02:07:22.177277 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20054,"thread_start_us":13490,"threads_started":1}
I20260929 02:07:22.177898 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.149840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.11.182.195:34689)
0929 02:07:22.168913 (+ 19073us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.168919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.169977 (+ 1058us) server_negotiation.cc:407] Connection header received
0929 02:07:22.170072 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.170079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.170277 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.170489 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:07:22.171428 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.172738 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.175310 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.176345 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.176570 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.176728 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.177273 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.177472 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18911,"thread_start_us":18756,"threads_started":1}
I20260929 02:07:22.179466 22456 tablet_service.cc:1467] Processing CreateTablet for tablet 293c3f60dd314806b3d238ed7c7f0a06 (DEFAULT_TABLE table=test-table [id=321b641d82f54bc686eb41242d42c86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:07:22.179687 22590 tablet_service.cc:1467] Processing CreateTablet for tablet 293c3f60dd314806b3d238ed7c7f0a06 (DEFAULT_TABLE table=test-table [id=321b641d82f54bc686eb41242d42c86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:07:22.180704 22456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293c3f60dd314806b3d238ed7c7f0a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:07:22.181213 22590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293c3f60dd314806b3d238ed7c7f0a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:07:22.181639 22723 tablet_service.cc:1467] Processing CreateTablet for tablet 293c3f60dd314806b3d238ed7c7f0a06 (DEFAULT_TABLE table=test-table [id=321b641d82f54bc686eb41242d42c86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:07:22.183133 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293c3f60dd314806b3d238ed7c7f0a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:07:22.191708 22806 tablet_bootstrap.cc:492] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: Bootstrap starting.
I20260929 02:07:22.194747 22806 tablet_bootstrap.cc:654] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: Neither blocks nor log segments found. Creating new log.
I20260929 02:07:22.195729 22806 log.cc:824] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: Log is configured to *not* fsync() on all Append() calls
I20260929 02:07:22.198127 22806 tablet_bootstrap.cc:492] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: No bootstrap required, opened a new log
I20260929 02:07:22.198395 22806 ts_tablet_manager.cc:1397] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 02:07:22.199791 22808 tablet_bootstrap.cc:492] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: Bootstrap starting.
I20260929 02:07:22.203349 22808 tablet_bootstrap.cc:654] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: Neither blocks nor log segments found. Creating new log.
I20260929 02:07:22.203514 22809 tablet_bootstrap.cc:492] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: Bootstrap starting.
I20260929 02:07:22.204660 22808 log.cc:824] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: Log is configured to *not* fsync() on all Append() calls
I20260929 02:07:22.206868 22809 tablet_bootstrap.cc:654] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: Neither blocks nor log segments found. Creating new log.
I20260929 02:07:22.208045 22809 log.cc:824] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:07:22.208166 22806 raft_consensus.cc:348] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } }
I20260929 02:07:22.208653 22806 raft_consensus.cc:374] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:07:22.208549 22808 tablet_bootstrap.cc:492] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: No bootstrap required, opened a new log
I20260929 02:07:22.208772 22806 raft_consensus.cc:729] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890aaebb0f4c48a3868073d3f0d72faa, State: Initialized, Role: FOLLOWER
I20260929 02:07:22.209110 22808 ts_tablet_manager.cc:1397] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:07:22.209470 22806 consensus_queue.cc:260] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } }
I20260929 02:07:22.211021 22806 ts_tablet_manager.cc:1428] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260929 02:07:22.211508 22809 tablet_bootstrap.cc:492] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: No bootstrap required, opened a new log
I20260929 02:07:22.211796 22809 ts_tablet_manager.cc:1397] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:07:22.219157 22808 raft_consensus.cc:348] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } }
I20260929 02:07:22.219553 22808 raft_consensus.cc:374] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:07:22.219720 22808 raft_consensus.cc:729] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c9bf4158fe3462698c284242064b599, State: Initialized, Role: FOLLOWER
I20260929 02:07:22.219877 22809 raft_consensus.cc:348] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } }
I20260929 02:07:22.220290 22809 raft_consensus.cc:374] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:07:22.220393 22809 raft_consensus.cc:729] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56599c2e473441e83d6632e57214f3c, State: Initialized, Role: FOLLOWER
I20260929 02:07:22.220333 22808 consensus_queue.cc:260] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } }
I20260929 02:07:22.220927 22809 consensus_queue.cc:260] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } }
I20260929 02:07:22.221544 22788 heartbeater.cc:503] Master 127.11.182.254:40849 was elected leader, sending a full tablet report...
I20260929 02:07:22.222007 22655 heartbeater.cc:503] Master 127.11.182.254:40849 was elected leader, sending a full tablet report...
I20260929 02:07:22.222208 22808 ts_tablet_manager.cc:1428] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: Time spent starting tablet: real 0.013s user 0.006s sys 0.005s
I20260929 02:07:22.222565 22809 ts_tablet_manager.cc:1428] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260929 02:07:22.327839 22522 tablet.cc:2369] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:07:22.329520 22789 tablet.cc:2369] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:07:22.335625 22656 tablet.cc:2369] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:07:22.394582 22814 raft_consensus.cc:482] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:07:22.394856 22814 raft_consensus.cc:504] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } }
I20260929 02:07:22.396466 22814 leader_election.cc:288] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 890aaebb0f4c48a3868073d3f0d72faa (127.11.182.193:37629), 2c9bf4158fe3462698c284242064b599 (127.11.182.195:34689)
I20260929 02:07:22.405400 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.396872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:48815 (local address 127.11.182.193:37629)
0929 02:07:22.397065 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.397073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.397619 (+ 546us) server_negotiation.cc:407] Connection header received
0929 02:07:22.397852 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.397860 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.397971 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.398071 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:07:22.398818 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.399988 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.403004 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.404253 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.404407 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.404481 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.405028 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.405151 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:07:22.405395 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.397106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:49887 (local address 127.11.182.195:34689)
0929 02:07:22.397337 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.397345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.397829 (+ 484us) server_negotiation.cc:407] Connection header received
0929 02:07:22.397967 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.397972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.398051 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.398133 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:07:22.398818 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.399990 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.402995 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.404242 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.404407 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.404485 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.405028 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.405151 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:07:22.405660 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.396930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:34689 (local address 127.11.182.194:49887)
0929 02:07:22.397727 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:22.397744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:22.397855 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:22.398148 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:22.398152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:22.398168 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:07:22.398671 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.398679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.400128 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.400132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.402790 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.402804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.404529 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.404535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.404774 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.404886 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 02:07:22.404979 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 02:07:22.405174 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":271,"thread_start_us":463,"threads_started":1}
I20260929 02:07:22.405700 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.396653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:37629 (local address 127.11.182.194:48815)
0929 02:07:22.397474 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 02:07:22.397501 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:07:22.397686 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:07:22.398116 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:07:22.398122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:07:22.398149 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:07:22.398644 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.398662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.400126 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.400132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.402790 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:07:22.402804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.404533 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.404538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:07:22.404778 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.404884 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 02:07:22.404979 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:07:22.405174 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":270,"thread_start_us":449,"threads_started":1}
I20260929 02:07:22.406628 22476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293c3f60dd314806b3d238ed7c7f0a06" candidate_uuid: "b56599c2e473441e83d6632e57214f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890aaebb0f4c48a3868073d3f0d72faa" is_pre_election: true
I20260929 02:07:22.406708 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293c3f60dd314806b3d238ed7c7f0a06" candidate_uuid: "b56599c2e473441e83d6632e57214f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c9bf4158fe3462698c284242064b599" is_pre_election: true
I20260929 02:07:22.407143 22476 raft_consensus.cc:2454] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b56599c2e473441e83d6632e57214f3c in term 0.
I20260929 02:07:22.407222 22743 raft_consensus.cc:2454] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b56599c2e473441e83d6632e57214f3c in term 0.
I20260929 02:07:22.407852 22546 leader_election.cc:302] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890aaebb0f4c48a3868073d3f0d72faa, b56599c2e473441e83d6632e57214f3c; no voters:
I20260929 02:07:22.408241 22814 raft_consensus.cc:2781] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:07:22.408383 22814 raft_consensus.cc:482] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:07:22.408486 22814 raft_consensus.cc:3037] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:07:22.410642 22814 raft_consensus.cc:504] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } }
I20260929 02:07:22.411320 22814 leader_election.cc:288] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [CANDIDATE]: Term 1 election: Requested vote from peers 890aaebb0f4c48a3868073d3f0d72faa (127.11.182.193:37629), 2c9bf4158fe3462698c284242064b599 (127.11.182.195:34689)
I20260929 02:07:22.411710 22476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293c3f60dd314806b3d238ed7c7f0a06" candidate_uuid: "b56599c2e473441e83d6632e57214f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890aaebb0f4c48a3868073d3f0d72faa"
I20260929 02:07:22.411914 22476 raft_consensus.cc:3037] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:07:22.412034 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293c3f60dd314806b3d238ed7c7f0a06" candidate_uuid: "b56599c2e473441e83d6632e57214f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c9bf4158fe3462698c284242064b599"
I20260929 02:07:22.412232 22743 raft_consensus.cc:3037] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:07:22.414206 22476 raft_consensus.cc:2454] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b56599c2e473441e83d6632e57214f3c in term 1.
I20260929 02:07:22.414527 22743 raft_consensus.cc:2454] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b56599c2e473441e83d6632e57214f3c in term 1.
I20260929 02:07:22.414615 22546 leader_election.cc:302] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890aaebb0f4c48a3868073d3f0d72faa, b56599c2e473441e83d6632e57214f3c; no voters:
I20260929 02:07:22.414932 22814 raft_consensus.cc:2781] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:07:22.415347 22814 raft_consensus.cc:686] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 1 LEADER]: Becoming Leader. State: Replica: b56599c2e473441e83d6632e57214f3c, State: Running, Role: LEADER
I20260929 02:07:22.415870 22814 consensus_queue.cc:237] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } }
I20260929 02:07:22.420249 22330 catalog_manager.cc:5495] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c reported cstate change: term changed from 0 to 1, leader changed from <none> to b56599c2e473441e83d6632e57214f3c (127.11.182.194). New cstate: current_term: 1 leader_uuid: "b56599c2e473441e83d6632e57214f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b56599c2e473441e83d6632e57214f3c" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 } health_report { overall_health: UNKNOWN } } }
I20260929 02:07:22.481390 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.475747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.11.182.194:40237)
0929 02:07:22.475951 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.475960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.476072 (+ 112us) server_negotiation.cc:407] Connection header received
0929 02:07:22.476320 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.476324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.476397 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.476578 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:07:22.476949 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.477671 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.478981 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.479640 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.479759 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.480212 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:07:22.480845 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:07:22.480985 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.481038 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.481131 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:07:22.489428 22743 raft_consensus.cc:1261] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [term 1 FOLLOWER]: Refusing update from remote peer b56599c2e473441e83d6632e57214f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:07:22.489428 22476 raft_consensus.cc:1261] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [term 1 FOLLOWER]: Refusing update from remote peer b56599c2e473441e83d6632e57214f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:07:22.490468 22817 consensus_queue.cc:1035] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c9bf4158fe3462698c284242064b599" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:07:22.490867 22814 consensus_queue.cc:1035] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "890aaebb0f4c48a3868073d3f0d72faa" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 37629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:07:22.502765 22822 mvcc.cc:205] Tried to move back new op lower bound from 7334492743625912320 to 7334492743339106304. Current Snapshot: MvccSnapshot[applied={T|T < 7334492743625912320}]
I20260929 02:07:22.515352 22330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:07:22.515611 22330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:07:22.518795 22330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcef0865afc64ab88d89ba470e331f56: Sending DeleteTablet for 3 replicas of tablet 293c3f60dd314806b3d238ed7c7f0a06
I20260929 02:07:22.519932 22590 tablet_service.cc:1514] Processing DeleteTablet for tablet 293c3f60dd314806b3d238ed7c7f0a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:07:22 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:37048
I20260929 02:07:22.520287 22456 tablet_service.cc:1514] Processing DeleteTablet for tablet 293c3f60dd314806b3d238ed7c7f0a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:07:22 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:35092
I20260929 02:07:22.520974 22723 tablet_service.cc:1514] Processing DeleteTablet for tablet 293c3f60dd314806b3d238ed7c7f0a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:07:22 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:33342
I20260929 02:07:22.521373 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 22393
I20260929 02:07:22.524076 22830 tablet_replica.cc:330] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: stopping tablet replica
I20260929 02:07:22.524554 22830 raft_consensus.cc:2229] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:07:22.525482 22829 tablet_replica.cc:330] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: stopping tablet replica
I20260929 02:07:22.525923 22829 raft_consensus.cc:2229] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:07:22.526628 22829 raft_consensus.cc:2258] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:07:22.527160 22830 raft_consensus.cc:2258] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:07:22.527844 22831 tablet_replica.cc:330] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: stopping tablet replica
I20260929 02:07:22.528278 22831 raft_consensus.cc:2229] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:07:22.528800 22831 raft_consensus.cc:2258] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:07:22.529062 22829 ts_tablet_manager.cc:1905] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:07:22.531565 22831 ts_tablet_manager.cc:1905] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:07:22.536553 22829 ts_tablet_manager.cc:1918] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:07:22.536796 22829 log.cc:1196] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/293c3f60dd314806b3d238ed7c7f0a06
I20260929 02:07:22.537374 22829 ts_tablet_manager.cc:1939] T 293c3f60dd314806b3d238ed7c7f0a06 P b56599c2e473441e83d6632e57214f3c: Deleting consensus metadata
I20260929 02:07:22.539011 22831 ts_tablet_manager.cc:1918] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:07:22.539171 22831 log.cc:1196] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/293c3f60dd314806b3d238ed7c7f0a06
I20260929 02:07:22.539645 22831 ts_tablet_manager.cc:1939] T 293c3f60dd314806b3d238ed7c7f0a06 P 2c9bf4158fe3462698c284242064b599: Deleting consensus metadata
I20260929 02:07:22.540505 22316 catalog_manager.cc:4841] TS b56599c2e473441e83d6632e57214f3c (127.11.182.194:40237): tablet 293c3f60dd314806b3d238ed7c7f0a06 (table test-table [id=321b641d82f54bc686eb41242d42c86e]) successfully deleted
I20260929 02:07:22.541924 22830 ts_tablet_manager.cc:1905] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:07:22.543835 22318 catalog_manager.cc:4841] TS 2c9bf4158fe3462698c284242064b599 (127.11.182.195:34689): tablet 293c3f60dd314806b3d238ed7c7f0a06 (table test-table [id=321b641d82f54bc686eb41242d42c86e]) successfully deleted
Sep 29 02:07:22 dist-test-slave-q06p krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:07:22.547673 22830 ts_tablet_manager.cc:1918] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:07:22.547842 22830 log.cc:1196] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/293c3f60dd314806b3d238ed7c7f0a06
I20260929 02:07:22.548364 22830 ts_tablet_manager.cc:1939] T 293c3f60dd314806b3d238ed7c7f0a06 P 890aaebb0f4c48a3868073d3f0d72faa: Deleting consensus metadata
I20260929 02:07:22.550079 22319 catalog_manager.cc:4841] TS 890aaebb0f4c48a3868073d3f0d72faa (127.11.182.193:37629): tablet 293c3f60dd314806b3d238ed7c7f0a06 (table test-table [id=321b641d82f54bc686eb41242d42c86e]) successfully deleted
I20260929 02:07:22.552824 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.522603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.11.182.193:37629)
0929 02:07:22.522828 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.522838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.535342 (+ 12504us) server_negotiation.cc:407] Connection header received
0929 02:07:22.535526 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.535533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.535622 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.535746 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:22.537418 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.538510 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.540438 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.541028 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.541159 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.546131 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:22.546144 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:22.546161 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:22.546212 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:22.550205 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.551037 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.551043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.551046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.551165 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.551577 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.551580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.551582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.551880 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:22.552103 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.552388 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.552530 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:07:22.553411 22516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:22.658080 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 22393
I20260929 02:07:22.681217 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 22527
Sep 29 02:07:22 dist-test-slave-q06p krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:07:22.698649 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.682001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.11.182.194:40237)
0929 02:07:22.682191 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.682196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.682220 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:07:22.682468 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.682473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.682537 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.682641 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:22.684220 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.685060 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.686994 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.687421 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.687548 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.692707 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:22.692722 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:22.692737 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:22.692772 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:22.695918 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.696638 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.696646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.696648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.696771 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.697132 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.697138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.697142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.697509 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:22.697840 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.698069 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.698280 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:07:22.699426 22650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:22.812570 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 22527
I20260929 02:07:22.831597 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 22660
Sep 29 02:07:22 dist-test-slave-q06p krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:07:22.846252 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:07:22.832479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.11.182.195:34689)
0929 02:07:22.832661 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:07:22.832669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:07:22.832712 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:07:22.832993 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:07:22.832998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:07:22.833090 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:07:22.833199 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:07:22.834460 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.835249 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.836434 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:07:22.836822 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:07:22.836954 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:07:22.841049 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:07:22.841064 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:07:22.841086 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:07:22.841119 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:07:22.843641 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.844271 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.844278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.844281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.844428 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:07:22.844798 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:07:22.844804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:07:22.844808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:07:22.845147 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:07:22.845432 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 02:07:22.845709 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 02:07:22.845924 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:07:22.846909 22783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:22.933952 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 22660
I20260929 02:07:22.950416 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 22300
I20260929 02:07:22.951570 22362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:07:23.059355 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 22300
2026-09-29T02:07:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4402 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:07:23 dist-test-slave-q06p krb5kdc[22839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:07:23 dist-test-slave-q06p krb5kdc[22839](info): set up 2 sockets
Sep 29 02:07:23 dist-test-slave-q06p krb5kdc[22839](info): commencing operation
krb5kdc: starting...
W20260929 02:07:24.168287 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:07:24 dist-test-slave-q06p krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:07:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.182.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.182.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/rangeradmin_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/rangeradmin_127.11.182.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.182.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.182.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/rangerlookup_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/rangerlookup_127.11.182.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.182.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.182.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/HTTP_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/HTTP_127.11.182.212@KRBTEST.COM.keytab.
I20260929 02:07:24.505666 11995 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskizkd1a/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/postgres -l logfile start
2026-09-29 02:07:27.019 UTC [22869] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:07:27.019 UTC [22869] LOG: listening on IPv4 address "127.11.182.212", port 37687
2026-09-29 02:07:27.023 UTC [22869] LOG: listening on Unix socket "/tmp/.s.PGSQL.37687"
2026-09-29 02:07:27.046 UTC [22872] LOG: database system was shut down at 2026-09-29 02:07:25 UTC
2026-09-29 02:07:27.049 UTC [22869] LOG: database system is ready to accept connections
I20260929 02:07:28.068216 11995 mini_postgres.cc:96] Postgres bound to 37687
2026-09-29 02:07:28.085 UTC [22880] FATAL: database "slave" does not exist
127.11.182.212:37687 - accepting connections
I20260929 02:07:28.086006 11995 mini_ranger.cc:161] Starting Ranger...
I20260929 02:07:28.111438 11995 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:07:28.614068 11995 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:07:28.614212 11995 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-admin
2026-09-29 02:07:28,897 [I] DB FLAVOR :POSTGRES
2026-09-29 02:07:28,898 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:07:28,898 [I] Checking connection..
2026-09-29 02:07:28,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:07:29,146 [I] Checking connection passed.
2026-09-29 02:07:29,146 [I] --------- Verifying version history table ---------
2026-09-29 02:07:29,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:07:29,394 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:07:29,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:07:29,623 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:07:29,623 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:07:29,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:07:29.880 UTC [22944] WARNING: there is no transaction in progress
2026-09-29 02:07:29,887 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:07:29,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:07:30,147 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:07:30,147 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:07:30,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:07:30,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:07:30,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q06p.c.gcp-upstream.internal','N') ;"
2026-09-29 02:07:30,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:07:31,142 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:07:31,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:07:31,406 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:07:31,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:07:31,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:07:31,911 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:07:31,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:07:33.000 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.022 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.044 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.065 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.087 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.110 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.173 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.181 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.198 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.210 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.221 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.236 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.247 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.254 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.262 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33.270 UTC [23052] WARNING: there is no transaction in progress
2026-09-29 02:07:33,969 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:07:33,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:07:34,219 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:07:34,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:07:34,459 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:07:34,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:07:34,721 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:07:34,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:07:34,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:07:35,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q06p.c.gcp-upstream.internal';"
2026-09-29 02:07:35,495 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:07:35,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:07:35,755 [I] DB_PATCHES have already been applied
I20260929 02:07:35.761113 11995 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-admin:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskizkd1a/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:07:35.761212 11995 mini_ranger.cc:193] Using host: 127.11.182.212
I20260929 02:07:35.776553 11995 mini_ranger.cc:227] Ranger admin URL: http://127.11.182.212:43615
Sep 29, 2026 2:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:07:36.928755 11995 mini_ranger.cc:160] Time spent starting Ranger: real 8.843s user 0.001s sys 0.039s
Sep 29, 2026 2:07:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:07:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 2:07:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:07:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:07:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:07:58.925242 11995 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.182.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.182.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/rangerkms_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/rangerkms_127.11.182.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.182.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.182.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.182.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/HTTP_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.182.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/HTTP_127.11.182.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:07:59 dist-test-slave-q06p krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647679, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:07:59.116962 11995 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:07:59.142529 11995 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:08:00.262342 11995 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:08:00.262506 11995 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-kms
I20260929 02:08:00.262646 11995 mini_ranger_kms.cc:227] Using postgres at 127.11.182.212:37687
2026-09-29 02:08:00,696 [I] DB FLAVOR :POSTGRES
2026-09-29 02:08:00,696 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:08:00,697 [I] Checking connection
2026-09-29 02:08:00,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:08:00,958 [I] connection success
2026-09-29 02:08:00,958 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:08:00,958 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:08:00,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:08:01,197 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:08:01,198 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:08:01,198 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:08:01,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:37687/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:08:01,462 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:08:01.966091 11995 mini_ranger_kms.cc:310] Created kms service
I20260929 02:08:02.122066 11995 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:08:02.167464 11995 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:08:02 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647679, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:08:02.558328 11995 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:08:02.558641 11995 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-kms:/tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:08:02.558707 11995 mini_ranger_kms.cc:240] Using host: 127.11.182.212
I20260929 02:08:02.576352 11995 mini_ranger_kms.cc:276] Ranger KMS PID: 23233
I20260929 02:08:02.576557 11995 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.182.212:45905
Sep 29, 2026 2:08:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:08:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:08:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:08:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:08:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:08:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 02:08:04 dist-test-slave-q06p krb5kdc[22839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647684, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.182.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:08:04.928182 11995 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.811s user 0.143s sys 0.311s
I20260929 02:08:04.928457 11995 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:08:04.928683 11995 mini_ranger_kms.cc:401] 127.11.182.212:45905/kms/v1/keys
Sep 29 02:08:05 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647684, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.182.212@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
Sep 29 02:08:05 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647684, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.182.212@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
Sep 29 02:08:07 dist-test-slave-q06p krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:08:07.927959 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:36997
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:36997
--ranger_config_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45905/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:08:08.269377 23283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:08:08.269768 23283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:08:08.269955 23283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:08:08.277616 23283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:08:08.277783 23283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:08:08.277892 23283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:08:08.277995 23283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:08:08.278083 23283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:08:08.288945 23283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.182.212:45905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:36997
--ranger_config_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:36997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:08:08.290256 23283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:08:08.291654 23283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:08:08.304164 23289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:08.304219 23291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:08.304240 23288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:08:08.304491 23283 server_base.cc:1034] running on GCE node
I20260929 02:08:08.305539 23283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:08:08.306932 23283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:08:08.308261 23283 hybrid_clock.cc:648] HybridClock initialized: now 1790647688308200 us; error 50 us; skew 500 ppm
I20260929 02:08:08.308635 23283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:08:08 dist-test-slave-q06p krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647688, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:08:08.317123 23283 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:08:08.318671 23283 webserver.cc:460] Webserver started at http://127.11.182.254:36587/ using document root <none> and password file <none>
I20260929 02:08:08.319407 23283 fs_manager.cc:359] Metadata directory not provided
I20260929 02:08:08.319537 23283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:08:08.319867 23283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:08:08 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647688, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:08:08.368458 23283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "af5dc83f647147ce9ac8d83e4b81972c"
format_stamp: "Formatted at 2026-09-29 02:08:08 on dist-test-slave-q06p"
server_key: "e531201f24fe5fc295a51cfc3b58f252"
server_key_iv: "d8e00366f9e9768a9333d8c5d4bb794a"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:08.369313 23283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "af5dc83f647147ce9ac8d83e4b81972c"
format_stamp: "Formatted at 2026-09-29 02:08:08 on dist-test-slave-q06p"
server_key: "e531201f24fe5fc295a51cfc3b58f252"
server_key_iv: "d8e00366f9e9768a9333d8c5d4bb794a"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:08.375401 23283 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260929 02:08:08.395938 23298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:08.397703 23283 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 02:08:08.397889 23283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "af5dc83f647147ce9ac8d83e4b81972c"
format_stamp: "Formatted at 2026-09-29 02:08:08 on dist-test-slave-q06p"
server_key: "e531201f24fe5fc295a51cfc3b58f252"
server_key_iv: "d8e00366f9e9768a9333d8c5d4bb794a"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:08.398144 23283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:08:08.418238 23283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:08:08.423292 23283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:08:08.423615 23283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:08:08.445591 23283 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:36997
I20260929 02:08:08.445647 23349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:36997 every 8 connection(s)
I20260929 02:08:08.447455 23283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:08:08.451570 23350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:08:08.457168 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 23283
I20260929 02:08:08.457387 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:08:08.458201 11995 external_mini_cluster.cc:1428] Setting key cf1b0a350ed475e8bf8f36d61172d878
I20260929 02:08:08.463011 23350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c: Bootstrap starting.
I20260929 02:08:08.466575 23350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c: Neither blocks nor log segments found. Creating new log.
I20260929 02:08:08.467922 23350 log.cc:824] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:08:08.471819 23350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c: No bootstrap required, opened a new log
Sep 29 02:08:08 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:08:08.479880 23350 raft_consensus.cc:348] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af5dc83f647147ce9ac8d83e4b81972c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36997 } }
I20260929 02:08:08.480278 23350 raft_consensus.cc:374] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:08:08.480424 23350 raft_consensus.cc:729] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5dc83f647147ce9ac8d83e4b81972c, State: Initialized, Role: FOLLOWER
I20260929 02:08:08.481079 23350 consensus_queue.cc:260] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af5dc83f647147ce9ac8d83e4b81972c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36997 } }
I20260929 02:08:08.481315 23350 raft_consensus.cc:388] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:08:08.481447 23350 raft_consensus.cc:482] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:08:08.481601 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:08.460510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.11.182.254:36997)
0929 02:08:08.461364 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:08.461374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:08.461417 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:08:08.463064 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:08.463085 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:08.463515 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:08.463809 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:08.465456 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:08.466676 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:08.468472 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:08.469931 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:08.470056 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:08.474471 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:08.474482 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:08.474497 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:08.474535 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:08.477517 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:08.478191 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:08.478207 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:08.478214 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:08.478362 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:08.478743 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:08.478750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:08.478753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:08.479372 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:08.479675 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:08.480088 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 02:08:08.480485 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":434,"threads_started":1}
I20260929 02:08:08.481639 23350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:08:08.485074 23350 raft_consensus.cc:504] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af5dc83f647147ce9ac8d83e4b81972c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36997 } }
I20260929 02:08:08.485605 23350 leader_election.cc:302] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af5dc83f647147ce9ac8d83e4b81972c; no voters:
I20260929 02:08:08.486227 23350 leader_election.cc:288] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:08:08.486342 23355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:08:08.487643 23355 raft_consensus.cc:686] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [term 1 LEADER]: Becoming Leader. State: Replica: af5dc83f647147ce9ac8d83e4b81972c, State: Running, Role: LEADER
I20260929 02:08:08.488159 23355 consensus_queue.cc:237] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af5dc83f647147ce9ac8d83e4b81972c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36997 } }
I20260929 02:08:08.488567 23350 sys_catalog.cc:564] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:08:08.491182 23357 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [sys.catalog]: SysCatalogTable state changed. Reason: New leader af5dc83f647147ce9ac8d83e4b81972c. Latest consensus state: current_term: 1 leader_uuid: "af5dc83f647147ce9ac8d83e4b81972c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af5dc83f647147ce9ac8d83e4b81972c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36997 } } }
I20260929 02:08:08.491580 23357 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [sys.catalog]: This master's current role is: LEADER
I20260929 02:08:08.492130 23356 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af5dc83f647147ce9ac8d83e4b81972c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af5dc83f647147ce9ac8d83e4b81972c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36997 } } }
I20260929 02:08:08.492487 23356 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c [sys.catalog]: This master's current role is: LEADER
I20260929 02:08:08.495078 23364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:08:08.500293 23364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:08:08.501019 23350 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:08:08.520010 23364 catalog_manager.cc:1347] Generated new cluster ID: 3e389658f041456096208fd9e826fc25
I20260929 02:08:08.520144 23364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:08:08.553480 23364 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:08:08.555681 23364 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:08:08.575641 23364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c: Generated new TSK 0
I20260929 02:08:08.576985 23364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:08:09.290195 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36997
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:08:09 dist-test-slave-q06p krb5kdc[22839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647689, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:08:09.693749 23395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:08:09.694114 23395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:08:09.694285 23395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:08:09.703670 23395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:08:09.703821 23395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:08:09.704001 23395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:08:09.714965 23395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.182.212:45905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:08:09.716636 23395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:08:09.718083 23395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:08:09.736876 23405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:09.737874 23406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:08:09.742587 23395 server_base.cc:1034] running on GCE node
W20260929 02:08:09.743288 23408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:08:09.744045 23395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:08:09.745087 23395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:08:09.746326 23395 hybrid_clock.cc:648] HybridClock initialized: now 1790647689746263 us; error 60 us; skew 500 ppm
I20260929 02:08:09.746658 23395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:08:09 dist-test-slave-q06p krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647689, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:08:09.751212 23395 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:08:09.753201 23395 webserver.cc:460] Webserver started at http://127.11.182.193:40755/ using document root <none> and password file <none>
I20260929 02:08:09.754166 23395 fs_manager.cc:359] Metadata directory not provided
I20260929 02:08:09.754330 23395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:08:09.754690 23395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:08:09 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647689, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:08:09.775849 23395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "0db7bfe06d9f44d4acaa9d90fff090e5"
format_stamp: "Formatted at 2026-09-29 02:08:09 on dist-test-slave-q06p"
server_key: "e8fcb70c88bb8d199fbf4f7e623770fa"
server_key_iv: "392079fcfb61e1f2ed60c08e662a19e2"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:09.776820 23395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "0db7bfe06d9f44d4acaa9d90fff090e5"
format_stamp: "Formatted at 2026-09-29 02:08:09 on dist-test-slave-q06p"
server_key: "e8fcb70c88bb8d199fbf4f7e623770fa"
server_key_iv: "392079fcfb61e1f2ed60c08e662a19e2"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:09.793759 23395 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.012s sys 0.005s
I20260929 02:08:09.819708 23415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:09.826807 23395 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.004s
I20260929 02:08:09.827140 23395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "0db7bfe06d9f44d4acaa9d90fff090e5"
format_stamp: "Formatted at 2026-09-29 02:08:09 on dist-test-slave-q06p"
server_key: "e8fcb70c88bb8d199fbf4f7e623770fa"
server_key_iv: "392079fcfb61e1f2ed60c08e662a19e2"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:09.827520 23395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:08:09.864951 23395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:08:09.870307 23395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:08:09.870674 23395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:08:09.871843 23395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:08:09.873383 23395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:08:09.873490 23395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:09.873639 23395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:08:09.873713 23395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:09.943382 23527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:38311 every 8 connection(s)
I20260929 02:08:09.943472 23395 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:38311
I20260929 02:08:09.946666 23395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:08:09.952491 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 23395
I20260929 02:08:09.952874 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:08:09.954442 11995 external_mini_cluster.cc:1428] Setting key c2d69d26a291a733b5956554481d5ad0
Sep 29 02:08:09 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647689, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
I20260929 02:08:09.998713 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:09.952616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36997 (local address 127.11.182.193:35933)
0929 02:08:09.960402 (+ 7786us) negotiation.cc:107] Waiting for socket to connect
0929 02:08:09.960492 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0929 02:08:09.964445 (+ 3953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:08:09.976608 (+ 12163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:08:09.976648 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:08:09.977502 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:08:09.978805 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:09.978864 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:09.980656 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:09.980662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:09.982911 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:09.982928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:09.984238 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:09.984246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:09.984567 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:09.985412 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:08:09.985450 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:08:09.988190 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:08:09.992915 (+ 4725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:08:09.992926 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:08:09.992946 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:08:09.994284 (+ 1338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:08:09.994789 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:08:09.994795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:08:09.994799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:08:09.995151 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:08:09.995907 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:08:09.995916 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:08:09.996364 (+ 448us) client_negotiation.cc:769] Sending connection context
0929 02:08:09.996658 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 02:08:09.997244 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7227,"thread_start_us":6719,"threads_started":1}
I20260929 02:08:09.999809 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:09.951089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:35933 (local address 127.11.182.254:36997)
0929 02:08:09.961614 (+ 10525us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:09.961621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:09.961642 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:08:09.964839 (+ 3197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:09.964849 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:09.964946 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:09.965073 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:09.979173 (+ 14100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:09.980422 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:09.983116 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:09.983788 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:09.983913 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:09.988468 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:09.988475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:09.988481 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:09.988522 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:09.992637 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:09.994465 (+ 1828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:09.994471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:09.994475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:09.994631 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:09.995298 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:09.995306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:09.995311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:09.995667 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:09.995829 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:09.999148 (+ 3319us) server_negotiation.cc:299] Negotiation successful
0929 02:08:09.999404 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10395,"thread_start_us":10209,"threads_started":1}
I20260929 02:08:10.002904 23528 heartbeater.cc:347] Connected to a master server at 127.11.182.254:36997
I20260929 02:08:10.003325 23528 heartbeater.cc:464] Registering TS with master...
I20260929 02:08:10.004379 23528 heartbeater.cc:511] Master 127.11.182.254:36997 requested a full tablet report, sending...
I20260929 02:08:10.007416 23315 ts_manager.cc:194] Registered new tserver with Master: 0db7bfe06d9f44d4acaa9d90fff090e5 (127.11.182.193:38311)
I20260929 02:08:10.009685 23315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:35933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:08:10.031265 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36997
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:08:10 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647689, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
Sep 29 02:08:10 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647689, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
W20260929 02:08:10.693857 23534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:08:10.694327 23534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:08:10.694519 23534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:08:10.705468 23534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:08:10.705694 23534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:08:10.705931 23534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:08:10.721318 23534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.182.212:45905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:08:10.723501 23534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:08:10.725288 23534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:08:10.761700 23541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:10.763643 23542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:10.773447 23545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:08:10.775494 23534 server_base.cc:1034] running on GCE node
I20260929 02:08:10.776332 23534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:08:10.787181 23534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:08:10.788563 23534 hybrid_clock.cc:648] HybridClock initialized: now 1790647690788463 us; error 102 us; skew 500 ppm
I20260929 02:08:10.789068 23534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:08:10 dist-test-slave-q06p krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647690, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:08:10.793540 23534 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:08:10.805806 23534 webserver.cc:460] Webserver started at http://127.11.182.194:45223/ using document root <none> and password file <none>
I20260929 02:08:10.806885 23534 fs_manager.cc:359] Metadata directory not provided
I20260929 02:08:10.807109 23534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:08:10.807536 23534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:08:10 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647690, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:08:10.829283 23534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "d561ef2eada649bd9b8d19b5529d67fb"
format_stamp: "Formatted at 2026-09-29 02:08:10 on dist-test-slave-q06p"
server_key: "1b223153308f0cd9b8a9dbaa1b1f265a"
server_key_iv: "2d1380c45cc5b74cc975798dcab960de"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:10.830462 23534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "d561ef2eada649bd9b8d19b5529d67fb"
format_stamp: "Formatted at 2026-09-29 02:08:10 on dist-test-slave-q06p"
server_key: "1b223153308f0cd9b8a9dbaa1b1f265a"
server_key_iv: "2d1380c45cc5b74cc975798dcab960de"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:10.837631 23534 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260929 02:08:10.873917 23551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:10.884249 23534 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.005s
I20260929 02:08:10.884506 23534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "d561ef2eada649bd9b8d19b5529d67fb"
format_stamp: "Formatted at 2026-09-29 02:08:10 on dist-test-slave-q06p"
server_key: "1b223153308f0cd9b8a9dbaa1b1f265a"
server_key_iv: "2d1380c45cc5b74cc975798dcab960de"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:10.884775 23534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:08:10.964888 23534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:08:10.981199 23534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:08:10.984611 23534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:08:10.999336 23534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:08:11.013948 23528 heartbeater.cc:503] Master 127.11.182.254:36997 was elected leader, sending a full tablet report...
I20260929 02:08:11.019845 23534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:08:11.020015 23534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:11.020174 23534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:08:11.020256 23534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:11.473184 23534 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:35953
I20260929 02:08:11.473196 23670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:35953 every 8 connection(s)
I20260929 02:08:11.474761 23534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:08:11.478686 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 23534
I20260929 02:08:11.478881 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:08:11.479387 11995 external_mini_cluster.cc:1428] Setting key 31081b791aa526f39283f18031350c70
Sep 29 02:08:11 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647690, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:08:11.499251 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:11.477448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:56891 (local address 127.11.182.254:36997)
0929 02:08:11.477999 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:11.478005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:11.479000 (+ 995us) server_negotiation.cc:407] Connection header received
0929 02:08:11.481555 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:11.481561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:11.481639 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:11.481759 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:11.483632 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:11.485222 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:11.487854 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:11.488621 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:11.488761 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:11.493265 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:11.493270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:11.493274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:11.493305 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:11.496127 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:11.496854 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:11.496859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:11.496867 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:11.496993 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:11.497432 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:11.497437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:11.497444 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:11.497708 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:11.497877 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:11.498584 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 02:08:11.498811 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":278,"threads_started":1}
I20260929 02:08:11.500270 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:11.477993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36997 (local address 127.11.182.194:56891)
0929 02:08:11.478796 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 02:08:11.478871 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 02:08:11.481240 (+ 2369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:08:11.482038 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:08:11.482051 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:08:11.482633 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:08:11.483435 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:11.483461 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:11.485424 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:11.485432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:11.487405 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:11.487422 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:11.488866 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:11.488875 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:11.489637 (+ 762us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:11.490469 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:08:11.490493 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:08:11.493024 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:08:11.496271 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:08:11.496280 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:08:11.496298 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:08:11.496716 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:08:11.497107 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:08:11.497113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:08:11.497117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:08:11.497300 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:08:11.497839 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:08:11.497848 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:08:11.498213 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 02:08:11.498486 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 02:08:11.499062 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":319,"threads_started":1}
I20260929 02:08:11.501936 23671 heartbeater.cc:347] Connected to a master server at 127.11.182.254:36997
I20260929 02:08:11.502251 23671 heartbeater.cc:464] Registering TS with master...
I20260929 02:08:11.503029 23671 heartbeater.cc:511] Master 127.11.182.254:36997 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:08:11.504688 23315 ts_manager.cc:194] Registered new tserver with Master: d561ef2eada649bd9b8d19b5529d67fb (127.11.182.194:35953)
I20260929 02:08:11.505522 23315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:56891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:08:11.535310 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36997
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:08:11.886726 23677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:08:11.887137 23677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:08:11.887310 23677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:08:11.896188 23677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:08:11.896337 23677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:08:11.896517 23677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:08:11.907015 23677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.182.212:45905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:08:11.908375 23677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:08:11.910084 23677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:08:11.923842 23685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:11.923995 23682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:11.923996 23683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:08:11.924436 23677 server_base.cc:1034] running on GCE node
I20260929 02:08:11.925016 23677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:08:11.925932 23677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:08:11.927130 23677 hybrid_clock.cc:648] HybridClock initialized: now 1790647691927061 us; error 53 us; skew 500 ppm
I20260929 02:08:11.927456 23677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:08:11 dist-test-slave-q06p krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647691, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:08:11.930804 23677 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:08:11.932312 23677 webserver.cc:460] Webserver started at http://127.11.182.195:38469/ using document root <none> and password file <none>
I20260929 02:08:11.932989 23677 fs_manager.cc:359] Metadata directory not provided
I20260929 02:08:11.933144 23677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:08:11.933404 23677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:08:11 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647691, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:08:11.951012 23677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "b69a0efca88c448cade00a0a8fad27bd"
format_stamp: "Formatted at 2026-09-29 02:08:11 on dist-test-slave-q06p"
server_key: "4b637f41fef073a939f327d38d5f0962"
server_key_iv: "52671a4b85d25a7d17cf5ddf9058de5d"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:11.952194 23677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "b69a0efca88c448cade00a0a8fad27bd"
format_stamp: "Formatted at 2026-09-29 02:08:11 on dist-test-slave-q06p"
server_key: "4b637f41fef073a939f327d38d5f0962"
server_key_iv: "52671a4b85d25a7d17cf5ddf9058de5d"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:11.958395 23677 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 02:08:11.975520 23692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:11.977108 23677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:08:11.977293 23677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "b69a0efca88c448cade00a0a8fad27bd"
format_stamp: "Formatted at 2026-09-29 02:08:11 on dist-test-slave-q06p"
server_key: "4b637f41fef073a939f327d38d5f0962"
server_key_iv: "52671a4b85d25a7d17cf5ddf9058de5d"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:11.977604 23677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:08:11.997514 23677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:08:12.001847 23677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:08:12.002156 23677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:08:12.003299 23677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:08:12.004809 23677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:08:12.004882 23677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:12.005045 23677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:08:12.005119 23677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:12.040890 23677 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:38009
I20260929 02:08:12.040930 23804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:38009 every 8 connection(s)
I20260929 02:08:12.042176 23677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:08:12.049525 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 23677
I20260929 02:08:12.049870 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:08:12.050400 11995 external_mini_cluster.cc:1428] Setting key 6149556bd4da598313d90df9a7752348
Sep 29 02:08:12 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647691, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:08:12.062346 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.044959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:49219 (local address 127.11.182.254:36997)
0929 02:08:12.045477 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.045481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.046662 (+ 1181us) server_negotiation.cc:407] Connection header received
0929 02:08:12.048426 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.048432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.048526 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.048674 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:12.050258 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.051079 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.052830 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.053359 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.053499 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.057106 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:12.057111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:12.057115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:12.057146 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:12.059752 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.060414 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.060418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.060421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.060544 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.060913 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.060916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.060918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.061165 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:12.061267 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.061908 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.062062 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":291,"threads_started":1}
I20260929 02:08:12.063346 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.045433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36997 (local address 127.11.182.195:49219)
0929 02:08:12.046489 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0929 02:08:12.046536 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:08:12.048155 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:08:12.048912 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:08:12.048920 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:08:12.049374 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:08:12.050089 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.050108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.051237 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.051243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.052673 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.052692 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.053799 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.053806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.054024 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.054712 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:08:12.054734 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:08:12.056860 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:08:12.059889 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:08:12.059898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:08:12.059912 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:08:12.060309 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:08:12.060637 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:08:12.060640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:08:12.060643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:08:12.060822 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:08:12.061265 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:08:12.061270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:08:12.061597 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 02:08:12.061820 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 02:08:12.062189 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":278,"threads_started":1}
I20260929 02:08:12.064848 23805 heartbeater.cc:347] Connected to a master server at 127.11.182.254:36997
I20260929 02:08:12.065151 23805 heartbeater.cc:464] Registering TS with master...
I20260929 02:08:12.066009 23805 heartbeater.cc:511] Master 127.11.182.254:36997 requested a full tablet report, sending...
I20260929 02:08:12.067404 23315 ts_manager.cc:194] Registered new tserver with Master: b69a0efca88c448cade00a0a8fad27bd (127.11.182.195:38009)
I20260929 02:08:12.068217 23315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:49219
I20260929 02:08:12.078936 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:08:12.097992 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.084404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.11.182.254:36997)
0929 02:08:12.084663 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.084673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.085217 (+ 544us) server_negotiation.cc:407] Connection header received
0929 02:08:12.085375 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.085379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.085445 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.085539 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:12.087042 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.087736 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.089154 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.089609 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.089709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.092538 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:12.092543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:12.092547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:12.092580 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:12.095326 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.096062 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.096067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.096069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.096199 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.096614 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.096618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.096620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.096949 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:12.097133 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.097547 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.097743 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:08:12.102780 23315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:08:12.110585 23315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:08:12.150933 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.124075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.11.182.195:38009)
0929 02:08:12.141109 (+ 17034us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.141116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.141142 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:08:12.141227 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.141233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.141470 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.141907 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:08:12.143147 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.144203 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.147843 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.149290 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.149468 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.150210 (+ 742us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.150307 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.150469 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16869,"thread_start_us":16723,"threads_started":1}
I20260929 02:08:12.151051 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.123304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:38009 (local address 127.0.0.1:38482)
0929 02:08:12.134375 (+ 11071us) negotiation.cc:107] Waiting for socket to connect
0929 02:08:12.134445 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 02:08:12.134719 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:08:12.141751 (+ 7032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:08:12.141760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:08:12.141818 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:08:12.142863 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.142885 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.144397 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.144402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.147631 (+ 3229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.147647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.149706 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.149714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.149981 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.150084 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 02:08:12.150233 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 02:08:12.150467 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10695,"mutex_wait_us":378,"spinlock_wait_cycles":10624,"thread_start_us":15969,"threads_started":1}
I20260929 02:08:12.151746 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.123620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38311 (local address 127.0.0.1:38672)
0929 02:08:12.139599 (+ 15979us) negotiation.cc:107] Waiting for socket to connect
0929 02:08:12.139623 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:08:12.139784 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:08:12.141751 (+ 1967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:08:12.141760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:08:12.141816 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:08:12.142373 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.142388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.144352 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.144360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.147326 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.147339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.149849 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.149855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.150893 (+ 1038us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.150961 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 02:08:12.151079 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 02:08:12.151256 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15817,"mutex_wait_us":1298,"spinlock_wait_cycles":6912,"thread_start_us":10361,"threads_started":1}
I20260929 02:08:12.152208 23822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.124291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:35953 (local address 127.0.0.1:60488)
0929 02:08:12.139977 (+ 15686us) negotiation.cc:107] Waiting for socket to connect
0929 02:08:12.140003 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:08:12.140609 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:08:12.142202 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:08:12.142208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:08:12.142243 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:08:12.142904 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.142916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.146354 (+ 3438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.146360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.148657 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.148669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.150671 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.150680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.150898 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.150983 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 02:08:12.151078 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:08:12.151219 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15499,"mutex_wait_us":167,"thread_start_us":15293,"threads_started":1}
I20260929 02:08:12.152218 23823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.124940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.11.182.194:35953)
0929 02:08:12.141497 (+ 16557us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.141508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.141541 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:08:12.141648 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.141656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.141931 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.142580 (+ 649us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:08:12.144461 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.146147 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.148870 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.150408 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.150582 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.150778 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.151719 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.151912 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16385,"thread_start_us":16270,"threads_started":1}
I20260929 02:08:12.153302 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.124289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.11.182.193:38311)
0929 02:08:12.140987 (+ 16698us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.140998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.141060 (+ 62us) server_negotiation.cc:407] Connection header received
0929 02:08:12.141180 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.141188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.141472 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.142593 (+ 1121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:08:12.142889 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.144188 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.147517 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.149262 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.149468 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.149676 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.151492 (+ 1816us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.152912 (+ 1420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16457,"thread_start_us":16297,"threads_started":1}
I20260929 02:08:12.156034 23740 tablet_service.cc:1467] Processing CreateTablet for tablet 2d058fa191f1482183f3071eb0068da9 (DEFAULT_TABLE table=test-table [id=d7e9896c1f874dc285d0c0e7541f7374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:08:12.156044 23606 tablet_service.cc:1467] Processing CreateTablet for tablet 2d058fa191f1482183f3071eb0068da9 (DEFAULT_TABLE table=test-table [id=d7e9896c1f874dc285d0c0e7541f7374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:08:12.156945 23463 tablet_service.cc:1467] Processing CreateTablet for tablet 2d058fa191f1482183f3071eb0068da9 (DEFAULT_TABLE table=test-table [id=d7e9896c1f874dc285d0c0e7541f7374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:08:12.157677 23606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d058fa191f1482183f3071eb0068da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:08:12.157677 23740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d058fa191f1482183f3071eb0068da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:08:12.158278 23463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d058fa191f1482183f3071eb0068da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:08:12.170908 23824 tablet_bootstrap.cc:492] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: Bootstrap starting.
I20260929 02:08:12.173695 23826 tablet_bootstrap.cc:492] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: Bootstrap starting.
I20260929 02:08:12.174528 23825 tablet_bootstrap.cc:492] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: Bootstrap starting.
I20260929 02:08:12.175064 23824 tablet_bootstrap.cc:654] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: Neither blocks nor log segments found. Creating new log.
I20260929 02:08:12.176671 23824 log.cc:824] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: Log is configured to *not* fsync() on all Append() calls
I20260929 02:08:12.178968 23826 tablet_bootstrap.cc:654] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: Neither blocks nor log segments found. Creating new log.
I20260929 02:08:12.179481 23825 tablet_bootstrap.cc:654] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: Neither blocks nor log segments found. Creating new log.
I20260929 02:08:12.180554 23826 log.cc:824] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:08:12.180935 23825 log.cc:824] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: Log is configured to *not* fsync() on all Append() calls
I20260929 02:08:12.182327 23824 tablet_bootstrap.cc:492] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: No bootstrap required, opened a new log
I20260929 02:08:12.182684 23824 ts_tablet_manager.cc:1397] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260929 02:08:12.183871 23826 tablet_bootstrap.cc:492] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: No bootstrap required, opened a new log
I20260929 02:08:12.184162 23826 ts_tablet_manager.cc:1397] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 02:08:12.189889 23825 tablet_bootstrap.cc:492] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: No bootstrap required, opened a new log
I20260929 02:08:12.190217 23825 ts_tablet_manager.cc:1397] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260929 02:08:12.192549 23826 raft_consensus.cc:348] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } }
I20260929 02:08:12.193121 23826 raft_consensus.cc:374] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:08:12.193284 23826 raft_consensus.cc:729] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d561ef2eada649bd9b8d19b5529d67fb, State: Initialized, Role: FOLLOWER
I20260929 02:08:12.193428 23824 raft_consensus.cc:348] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } }
I20260929 02:08:12.193946 23824 raft_consensus.cc:374] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:08:12.194054 23824 raft_consensus.cc:729] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db7bfe06d9f44d4acaa9d90fff090e5, State: Initialized, Role: FOLLOWER
I20260929 02:08:12.194069 23826 consensus_queue.cc:260] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } }
I20260929 02:08:12.194734 23824 consensus_queue.cc:260] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } }
I20260929 02:08:12.195546 23671 heartbeater.cc:503] Master 127.11.182.254:36997 was elected leader, sending a full tablet report...
I20260929 02:08:12.197580 23826 ts_tablet_manager.cc:1428] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260929 02:08:12.199518 23824 ts_tablet_manager.cc:1428] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: Time spent starting tablet: real 0.017s user 0.012s sys 0.007s
W20260929 02:08:12.201176 23529 tablet.cc:2369] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:08:12.203446 23825 raft_consensus.cc:348] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } }
I20260929 02:08:12.203964 23825 raft_consensus.cc:374] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:08:12.204066 23825 raft_consensus.cc:729] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69a0efca88c448cade00a0a8fad27bd, State: Initialized, Role: FOLLOWER
I20260929 02:08:12.204679 23825 consensus_queue.cc:260] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } }
I20260929 02:08:12.205708 23805 heartbeater.cc:503] Master 127.11.182.254:36997 was elected leader, sending a full tablet report...
I20260929 02:08:12.206394 23825 ts_tablet_manager.cc:1428] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260929 02:08:12.227169 23672 tablet.cc:2369] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:08:12.254523 23830 raft_consensus.cc:482] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:08:12.254835 23830 raft_consensus.cc:504] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } }
I20260929 02:08:12.256733 23830 leader_election.cc:288] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b69a0efca88c448cade00a0a8fad27bd (127.11.182.195:38009), 0db7bfe06d9f44d4acaa9d90fff090e5 (127.11.182.193:38311)
I20260929 02:08:12.263670 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.257221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:58265 (local address 127.11.182.193:38311)
0929 02:08:12.257455 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.257462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.258189 (+ 727us) server_negotiation.cc:407] Connection header received
0929 02:08:12.258383 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.258391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.258483 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.258595 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:08:12.259323 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.260050 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.261892 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.262646 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.262875 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.262955 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.263257 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.263409 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:08:12.263700 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.257252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38311 (local address 127.11.182.194:58265)
0929 02:08:12.257997 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 02:08:12.258020 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:08:12.258210 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:08:12.258667 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:08:12.258674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:08:12.258709 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:08:12.259154 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.259163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.260251 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.260259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.261721 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.261732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.262890 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.262896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.263055 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.263087 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:08:12.263189 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 02:08:12.263334 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":809,"threads_started":1}
I20260929 02:08:12.264809 23834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.257858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:38009 (local address 127.11.182.194:34473)
0929 02:08:12.258695 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 02:08:12.258713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:08:12.259027 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:08:12.259394 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:08:12.259398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:08:12.259421 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:08:12.259749 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.259757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.260766 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.260771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.262114 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:12.262124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.264187 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.264196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:12.264361 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.264384 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:08:12.264460 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 02:08:12.264544 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":7,"spinlock_wait_cycles":25472,"thread_start_us":472,"threads_started":1}
I20260929 02:08:12.264931 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.257295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:34473 (local address 127.11.182.195:38009)
0929 02:08:12.257499 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.257506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.258798 (+ 1292us) server_negotiation.cc:407] Connection header received
0929 02:08:12.259189 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.259194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.259267 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.259371 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:08:12.259907 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.260611 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.262282 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.263408 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.263556 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.263627 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.264537 (+ 910us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.264674 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:08:12.264892 23483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d058fa191f1482183f3071eb0068da9" candidate_uuid: "d561ef2eada649bd9b8d19b5529d67fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" is_pre_election: true
I20260929 02:08:12.265525 23483 raft_consensus.cc:2454] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d561ef2eada649bd9b8d19b5529d67fb in term 0.
I20260929 02:08:12.266294 23552 leader_election.cc:302] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db7bfe06d9f44d4acaa9d90fff090e5, d561ef2eada649bd9b8d19b5529d67fb; no voters:
I20260929 02:08:12.266402 23760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d058fa191f1482183f3071eb0068da9" candidate_uuid: "d561ef2eada649bd9b8d19b5529d67fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69a0efca88c448cade00a0a8fad27bd" is_pre_election: true
I20260929 02:08:12.266805 23830 raft_consensus.cc:2781] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:08:12.266999 23830 raft_consensus.cc:482] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:08:12.267118 23830 raft_consensus.cc:3037] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:08:12.267035 23760 raft_consensus.cc:2454] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d561ef2eada649bd9b8d19b5529d67fb in term 0.
I20260929 02:08:12.269559 23830 raft_consensus.cc:504] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } }
I20260929 02:08:12.270311 23830 leader_election.cc:288] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [CANDIDATE]: Term 1 election: Requested vote from peers b69a0efca88c448cade00a0a8fad27bd (127.11.182.195:38009), 0db7bfe06d9f44d4acaa9d90fff090e5 (127.11.182.193:38311)
I20260929 02:08:12.270833 23760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d058fa191f1482183f3071eb0068da9" candidate_uuid: "d561ef2eada649bd9b8d19b5529d67fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69a0efca88c448cade00a0a8fad27bd"
I20260929 02:08:12.271075 23760 raft_consensus.cc:3037] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:08:12.270992 23483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d058fa191f1482183f3071eb0068da9" candidate_uuid: "d561ef2eada649bd9b8d19b5529d67fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5"
I20260929 02:08:12.271232 23483 raft_consensus.cc:3037] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:08:12.273219 23760 raft_consensus.cc:2454] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d561ef2eada649bd9b8d19b5529d67fb in term 1.
I20260929 02:08:12.273476 23483 raft_consensus.cc:2454] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d561ef2eada649bd9b8d19b5529d67fb in term 1.
I20260929 02:08:12.273677 23553 leader_election.cc:302] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b69a0efca88c448cade00a0a8fad27bd, d561ef2eada649bd9b8d19b5529d67fb; no voters:
I20260929 02:08:12.274039 23830 raft_consensus.cc:2781] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:08:12.274545 23830 raft_consensus.cc:686] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 1 LEADER]: Becoming Leader. State: Replica: d561ef2eada649bd9b8d19b5529d67fb, State: Running, Role: LEADER
I20260929 02:08:12.275116 23830 consensus_queue.cc:237] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } }
I20260929 02:08:12.280098 23315 catalog_manager.cc:5495] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb reported cstate change: term changed from 0 to 1, leader changed from <none> to d561ef2eada649bd9b8d19b5529d67fb (127.11.182.194). New cstate: current_term: 1 leader_uuid: "d561ef2eada649bd9b8d19b5529d67fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d561ef2eada649bd9b8d19b5529d67fb" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 35953 } health_report { overall_health: HEALTHY } } }
W20260929 02:08:12.293871 23806 tablet.cc:2369] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:08:12.307520 23823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.301832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.11.182.194:35953)
0929 02:08:12.302081 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.302086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.302106 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:08:12.302338 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.302343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.302442 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.302601 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:08:12.303051 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.303770 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.305273 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.305674 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.305820 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.306309 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:08:12.306818 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:08:12.307046 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.307132 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.307213 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:08:12.315573 23483 raft_consensus.cc:1261] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [term 1 FOLLOWER]: Refusing update from remote peer d561ef2eada649bd9b8d19b5529d67fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:08:12.315642 23760 raft_consensus.cc:1261] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [term 1 FOLLOWER]: Refusing update from remote peer d561ef2eada649bd9b8d19b5529d67fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:08:12.316586 23835 consensus_queue.cc:1035] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db7bfe06d9f44d4acaa9d90fff090e5" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:08:12.316943 23830 consensus_queue.cc:1035] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69a0efca88c448cade00a0a8fad27bd" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 38009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:08:12.344695 23313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:08:12.344962 23313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:08:12.349095 23313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af5dc83f647147ce9ac8d83e4b81972c: Sending DeleteTablet for 3 replicas of tablet 2d058fa191f1482183f3071eb0068da9
I20260929 02:08:12.350471 23740 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d058fa191f1482183f3071eb0068da9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:08:12 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:38482
I20260929 02:08:12.350519 23606 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d058fa191f1482183f3071eb0068da9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:08:12 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:60488
I20260929 02:08:12.351881 23849 tablet_replica.cc:330] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: stopping tablet replica
I20260929 02:08:12.351878 23848 tablet_replica.cc:330] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: stopping tablet replica
I20260929 02:08:12.351981 23463 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d058fa191f1482183f3071eb0068da9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:08:12 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:38672
I20260929 02:08:12.352337 23849 raft_consensus.cc:2229] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:08:12.352581 23848 raft_consensus.cc:2229] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:08:12.352852 23849 raft_consensus.cc:2258] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:08:12.353358 23848 raft_consensus.cc:2258] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:08:12.354640 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 23395
I20260929 02:08:12.355028 23849 ts_tablet_manager.cc:1905] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:08:12.355592 23848 ts_tablet_manager.cc:1905] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:08:12.362169 23849 ts_tablet_manager.cc:1918] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:08:12.362352 23849 log.cc:1196] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/2d058fa191f1482183f3071eb0068da9
I20260929 02:08:12.362922 23849 ts_tablet_manager.cc:1939] T 2d058fa191f1482183f3071eb0068da9 P b69a0efca88c448cade00a0a8fad27bd: Deleting consensus metadata
I20260929 02:08:12.364804 23848 ts_tablet_manager.cc:1918] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:08:12.365003 23848 log.cc:1196] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/2d058fa191f1482183f3071eb0068da9
I20260929 02:08:12.365024 23300 catalog_manager.cc:4841] TS b69a0efca88c448cade00a0a8fad27bd (127.11.182.195:38009): tablet 2d058fa191f1482183f3071eb0068da9 (table test-table [id=d7e9896c1f874dc285d0c0e7541f7374]) successfully deleted
I20260929 02:08:12.365605 23848 ts_tablet_manager.cc:1939] T 2d058fa191f1482183f3071eb0068da9 P d561ef2eada649bd9b8d19b5529d67fb: Deleting consensus metadata
I20260929 02:08:12.367731 23302 catalog_manager.cc:4841] TS d561ef2eada649bd9b8d19b5529d67fb (127.11.182.194:35953): tablet 2d058fa191f1482183f3071eb0068da9 (table test-table [id=d7e9896c1f874dc285d0c0e7541f7374]) successfully deleted
I20260929 02:08:12.372092 23850 tablet_replica.cc:330] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: stopping tablet replica
I20260929 02:08:12.372545 23850 raft_consensus.cc:2229] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:08:12.373047 23850 raft_consensus.cc:2258] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:08:12.375960 23850 ts_tablet_manager.cc:1905] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:08:12 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:08:12.381911 23850 ts_tablet_manager.cc:1918] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:08:12.382089 23850 log.cc:1196] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/2d058fa191f1482183f3071eb0068da9
I20260929 02:08:12.382763 23850 ts_tablet_manager.cc:1939] T 2d058fa191f1482183f3071eb0068da9 P 0db7bfe06d9f44d4acaa9d90fff090e5: Deleting consensus metadata
I20260929 02:08:12.384531 23299 catalog_manager.cc:4841] TS 0db7bfe06d9f44d4acaa9d90fff090e5 (127.11.182.193:38311): tablet 2d058fa191f1482183f3071eb0068da9 (table test-table [id=d7e9896c1f874dc285d0c0e7541f7374]) successfully deleted
I20260929 02:08:12.391108 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.357427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.11.182.193:38311)
0929 02:08:12.357634 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.357641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.371506 (+ 13865us) server_negotiation.cc:407] Connection header received
0929 02:08:12.371733 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.371740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.371839 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.371927 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:12.373752 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.374594 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.376597 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.377268 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.377445 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.383105 (+ 5660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:12.383121 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:12.383140 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:12.383191 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:12.387552 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.388301 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.388310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.388314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.388473 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.388964 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.388969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.388973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.389501 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:12.389850 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.390240 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.390500 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:08:12.391867 23523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:08:12.514750 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 23395
I20260929 02:08:12.533665 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 23534
Sep 29 02:08:12 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:08:12.550416 23823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.534489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.11.182.194:35953)
0929 02:08:12.534695 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.534701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.534779 (+ 78us) server_negotiation.cc:407] Connection header received
0929 02:08:12.535045 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.535053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.535152 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.535255 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:12.536572 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.537414 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.538697 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.539254 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.539410 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.543771 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:12.543798 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:12.543824 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:12.543869 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:12.547518 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.548268 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.548277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.548280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.548429 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.548813 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.548819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.548822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.549195 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:12.549527 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.549823 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.550044 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:08:12.551105 23666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:08:12.672277 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 23534
I20260929 02:08:12.691942 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 23677
Sep 29 02:08:12 dist-test-slave-q06p krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:08:12.708817 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:12.692803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.11.182.195:38009)
0929 02:08:12.692989 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:12.692996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:12.693103 (+ 107us) server_negotiation.cc:407] Connection header received
0929 02:08:12.693366 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:12.693370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:12.693441 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:12.693566 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:12.695069 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.695897 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.697218 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:12.697652 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:12.697823 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:12.702506 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:12.702522 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:12.702538 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:12.702577 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:12.705807 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.706578 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.706586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.706588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.706741 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:12.707191 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:12.707195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:12.707197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:12.707687 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:12.707989 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:12.708389 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 02:08:12.708560 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:08:12.709530 23800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:08:12.799973 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 23677
I20260929 02:08:12.816295 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 23283
I20260929 02:08:12.817560 23345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:08:12.931277 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 23283
I20260929 02:08:12.956284 11995 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:08:13.291965 11995 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:08:13.292189 11995 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:08:13.637768 11995 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:08:13.638 UTC [22869] LOG: received smart shutdown request
2026-09-29 02:08:13.642 UTC [22869] LOG: background worker "logical replication launcher" (PID 22878) exited with exit code 1
2026-09-29 02:08:13.642 UTC [22873] LOG: shutting down
2026-09-29 02:08:13.834 UTC [22869] LOG: database system is shut down
2026-09-29T02:08:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50899 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:08:14 dist-test-slave-q06p krb5kdc[23864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:08:14 dist-test-slave-q06p krb5kdc[23864](info): set up 2 sockets
Sep 29 02:08:14 dist-test-slave-q06p krb5kdc[23864](info): commencing operation
krb5kdc: starting...
W20260929 02:08:15.079690 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:08:15 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:08:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.182.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.182.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/rangeradmin_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/rangeradmin_127.11.182.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.182.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.182.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/rangerlookup_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/rangerlookup_127.11.182.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.182.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.182.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/HTTP_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/HTTP_127.11.182.212@KRBTEST.COM.keytab.
I20260929 02:08:15.418215 11995 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskizkd1a/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/postgres -l logfile start
2026-09-29 02:08:18.079 UTC [23894] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:08:18.079 UTC [23894] LOG: listening on IPv4 address "127.11.182.212", port 42877
2026-09-29 02:08:18.083 UTC [23894] LOG: listening on Unix socket "/tmp/.s.PGSQL.42877"
2026-09-29 02:08:18.105 UTC [23897] LOG: database system was shut down at 2026-09-29 02:08:16 UTC
2026-09-29 02:08:18.110 UTC [23894] LOG: database system is ready to accept connections
I20260929 02:08:19.119364 11995 mini_postgres.cc:96] Postgres bound to 42877
2026-09-29 02:08:19.136 UTC [23905] FATAL: database "slave" does not exist
127.11.182.212:42877 - accepting connections
I20260929 02:08:19.137362 11995 mini_ranger.cc:161] Starting Ranger...
I20260929 02:08:19.166725 11995 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:08:19.711820 11995 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:08:19.711942 11995 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-admin
2026-09-29 02:08:20,030 [I] DB FLAVOR :POSTGRES
2026-09-29 02:08:20,030 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:08:20,030 [I] Checking connection..
2026-09-29 02:08:20,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:08:20,270 [I] Checking connection passed.
2026-09-29 02:08:20,270 [I] --------- Verifying version history table ---------
2026-09-29 02:08:20,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:08:20,526 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:08:20,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:08:20,785 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:08:20,785 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:08:20,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:08:21.042 UTC [23969] WARNING: there is no transaction in progress
2026-09-29 02:08:21,049 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:08:21,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:08:21,297 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:08:21,298 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:08:21,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:08:21,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:08:21,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q06p.c.gcp-upstream.internal','N') ;"
2026-09-29 02:08:22,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:08:22,297 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:08:22,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:08:22,527 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:08:22,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:08:22,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:08:22,998 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:08:22,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:08:23.937 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:23.953 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:23.968 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:23.983 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:23.998 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.016 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.078 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.085 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.100 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.110 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.121 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.134 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.144 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.151 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.157 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24.164 UTC [24077] WARNING: there is no transaction in progress
2026-09-29 02:08:24,775 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:08:24,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:08:25,023 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:08:25,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:08:25,279 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:08:25,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:08:25,518 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:08:25,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:08:25,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:08:26,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q06p.c.gcp-upstream.internal';"
2026-09-29 02:08:26,243 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:08:26,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:08:26,499 [I] DB_PATCHES have already been applied
I20260929 02:08:26.504933 11995 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-admin:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskizkd1a/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:08:26.505023 11995 mini_ranger.cc:193] Using host: 127.11.182.212
I20260929 02:08:26.519275 11995 mini_ranger.cc:227] Ranger admin URL: http://127.11.182.212:39717
Sep 29, 2026 2:08:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:08:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:08:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:08:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:08:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:08:27.661357 11995 mini_ranger.cc:160] Time spent starting Ranger: real 8.524s user 0.000s sys 0.038s
Sep 29, 2026 2:08:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:08:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:08:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:08:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:08:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:08:48.758509 11995 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.182.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.182.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/rangerkms_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.182.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/rangerkms_127.11.182.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.182.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.182.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.182.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/HTTP_127.11.182.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.182.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/HTTP_127.11.182.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:08:48 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647728, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:08:48.933247 11995 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:08:48.954094 11995 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:08:49.806893 11995 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:08:49.807005 11995 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-kms
I20260929 02:08:49.807066 11995 mini_ranger_kms.cc:227] Using postgres at 127.11.182.212:42877
2026-09-29 02:08:50,147 [I] DB FLAVOR :POSTGRES
2026-09-29 02:08:50,148 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:08:50,148 [I] Checking connection
2026-09-29 02:08:50,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:08:50,371 [I] connection success
2026-09-29 02:08:50,371 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:08:50,371 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:08:50,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:08:50,595 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:08:50,595 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:08:50,595 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:08:50,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.182.212:42877/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:08:50,846 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:08:51.336040 11995 mini_ranger_kms.cc:310] Created kms service
I20260929 02:08:51.537592 11995 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:08:51.583133 11995 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:08:51 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647728, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:08:51.983541 11995 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:08:51.984010 11995 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-kms:/tmp/dist-test-taskizkd1a/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskizkd1a/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:08:51.984087 11995 mini_ranger_kms.cc:240] Using host: 127.11.182.212
I20260929 02:08:52.001238 11995 mini_ranger_kms.cc:276] Ranger KMS PID: 24257
I20260929 02:08:52.001359 11995 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.182.212:45647
Sep 29, 2026 2:08:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:08:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:08:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:08:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:08:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:08:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 02:08:54.351111 11995 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.418s user 0.055s sys 0.304s
I20260929 02:08:54.351361 11995 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:08:54.351572 11995 mini_ranger_kms.cc:401] 127.11.182.212:45647/kms/v1/keys
Sep 29 02:08:54 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.182.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:08:55 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.182.212@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
Sep 29 02:08:55 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.182.212@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
Sep 29 02:08:57 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:08:57.679234 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33067
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:33067
--ranger_config_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45647/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:08:57.997356 24308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:08:57.997632 24308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:08:57.997776 24308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:08:58.005007 24308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:08:58.005107 24308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:08:58.005187 24308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:08:58.005285 24308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:08:58.005369 24308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:08:58.015559 24308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.182.212:45647/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:33067
--ranger_config_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:08:58.016872 24308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:08:58.018139 24308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:08:58.031579 24314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:58.031608 24313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:58.031651 24316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:08:58.033059 24308 server_base.cc:1034] running on GCE node
I20260929 02:08:58.033838 24308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:08:58.035387 24308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:08:58.036603 24308 hybrid_clock.cc:648] HybridClock initialized: now 1790647738036522 us; error 78 us; skew 500 ppm
I20260929 02:08:58.036945 24308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:08:58 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647738, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:08:58.041703 24308 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:08:58.043409 24308 webserver.cc:460] Webserver started at http://127.11.182.254:34241/ using document root <none> and password file <none>
I20260929 02:08:58.044078 24308 fs_manager.cc:359] Metadata directory not provided
I20260929 02:08:58.044171 24308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:08:58.044526 24308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:08:58 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647738, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:08:58.093004 24308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "788c52b5896648f7be8f55d8dc9f6b11"
format_stamp: "Formatted at 2026-09-29 02:08:58 on dist-test-slave-q06p"
server_key: "63655c8ed2b144b97bc0dd61a96218ff"
server_key_iv: "fe93f040466a124a71324be60546bfe5"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:58.094080 24308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "788c52b5896648f7be8f55d8dc9f6b11"
format_stamp: "Formatted at 2026-09-29 02:08:58 on dist-test-slave-q06p"
server_key: "63655c8ed2b144b97bc0dd61a96218ff"
server_key_iv: "fe93f040466a124a71324be60546bfe5"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:58.100988 24308 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260929 02:08:58.120642 24323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:58.122393 24308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 02:08:58.122630 24308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "788c52b5896648f7be8f55d8dc9f6b11"
format_stamp: "Formatted at 2026-09-29 02:08:58 on dist-test-slave-q06p"
server_key: "63655c8ed2b144b97bc0dd61a96218ff"
server_key_iv: "fe93f040466a124a71324be60546bfe5"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:58.122917 24308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:08:58.139189 24308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:08:58.143873 24308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:08:58.144192 24308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:08:58.163232 24308 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:33067
I20260929 02:08:58.163256 24374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:33067 every 8 connection(s)
I20260929 02:08:58.164961 24308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:08:58.169132 24375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:08:58.171989 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 24308
I20260929 02:08:58.172194 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:08:58.173035 11995 external_mini_cluster.cc:1428] Setting key 494f76a4f89b6e9351eaf74b834832d5
I20260929 02:08:58.185029 24375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Bootstrap starting.
I20260929 02:08:58.188694 24375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Neither blocks nor log segments found. Creating new log.
I20260929 02:08:58.189749 24375 log.cc:824] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Log is configured to *not* fsync() on all Append() calls
I20260929 02:08:58.192919 24375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: No bootstrap required, opened a new log
Sep 29 02:08:58 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:08:58.203009 24375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } }
I20260929 02:08:58.203402 24375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:08:58.203498 24375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788c52b5896648f7be8f55d8dc9f6b11, State: Initialized, Role: FOLLOWER
I20260929 02:08:58.204123 24375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } }
I20260929 02:08:58.204404 24375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:08:58.204559 24375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:08:58.204735 24375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:08:58.207471 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:58.175185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.11.182.254:33067)
0929 02:08:58.183438 (+ 8253us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:58.183454 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:58.183827 (+ 373us) server_negotiation.cc:407] Connection header received
0929 02:08:58.186285 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:58.186310 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:58.186898 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:58.187345 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:58.188905 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:58.190992 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:58.192795 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:58.193804 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:58.193970 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:58.198782 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:58.198793 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:58.198816 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:58.198884 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:58.202813 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:58.203645 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:58.203654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:58.203663 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:58.203832 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:58.204235 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:58.204241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:58.204244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:58.205018 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:58.205379 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:58.205730 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 02:08:58.206133 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8000,"thread_start_us":7784,"threads_started":1}
I20260929 02:08:58.208011 24375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } }
I20260929 02:08:58.208556 24375 leader_election.cc:302] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788c52b5896648f7be8f55d8dc9f6b11; no voters:
I20260929 02:08:58.209210 24375 leader_election.cc:288] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:08:58.209332 24380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:08:58.215706 24380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 1 LEADER]: Becoming Leader. State: Replica: 788c52b5896648f7be8f55d8dc9f6b11, State: Running, Role: LEADER
I20260929 02:08:58.216305 24380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } }
I20260929 02:08:58.216668 24375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:08:58.224777 24381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788c52b5896648f7be8f55d8dc9f6b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } } }
I20260929 02:08:58.225114 24381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: This master's current role is: LEADER
I20260929 02:08:58.224494 24382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788c52b5896648f7be8f55d8dc9f6b11. Latest consensus state: current_term: 1 leader_uuid: "788c52b5896648f7be8f55d8dc9f6b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } } }
I20260929 02:08:58.225811 24382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: This master's current role is: LEADER
I20260929 02:08:58.235421 24387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:08:58.243000 24387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:08:58.266238 24387 catalog_manager.cc:1347] Generated new cluster ID: 2e769a7298914e19bb89d9d497f7b763
I20260929 02:08:58.266394 24387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:08:58.275043 24375 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:08:58.295264 24387 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:08:58.297725 24387 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:08:58.306157 24387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Generated new TSK 0
I20260929 02:08:58.307413 24387 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:08:59.071030 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45647/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33067
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:08:59 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647739, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:08:59.500656 24421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:08:59.501178 24421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:08:59.501371 24421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:08:59.512902 24421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:08:59.513053 24421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:08:59.513300 24421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:08:59.529065 24421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.182.212:45647/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:08:59.530709 24421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:08:59.532403 24421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:08:59.568893 24431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:59.568949 24432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:08:59.576157 24434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:08:59.576738 24421 server_base.cc:1034] running on GCE node
I20260929 02:08:59.577587 24421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:08:59.578666 24421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:08:59.579980 24421 hybrid_clock.cc:648] HybridClock initialized: now 1790647739579915 us; error 54 us; skew 500 ppm
I20260929 02:08:59.580374 24421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:08:59 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647739, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:08:59.583710 24421 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:08:59.585553 24421 webserver.cc:460] Webserver started at http://127.11.182.193:35741/ using document root <none> and password file <none>
I20260929 02:08:59.586257 24421 fs_manager.cc:359] Metadata directory not provided
I20260929 02:08:59.586397 24421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:08:59.586714 24421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:08:59 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647739, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:08:59.605321 24421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "cedd592820f84c90b03484926a0c18c6"
format_stamp: "Formatted at 2026-09-29 02:08:59 on dist-test-slave-q06p"
server_key: "94e42ec9a2170929b90d614ebe83e936"
server_key_iv: "098f1dc345e838f2deceabc1470c7574"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:59.606184 24421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "cedd592820f84c90b03484926a0c18c6"
format_stamp: "Formatted at 2026-09-29 02:08:59 on dist-test-slave-q06p"
server_key: "94e42ec9a2170929b90d614ebe83e936"
server_key_iv: "098f1dc345e838f2deceabc1470c7574"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:59.632834 24421 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.008s sys 0.016s
I20260929 02:08:59.665418 24441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:59.667631 24421 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:08:59.667842 24421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "cedd592820f84c90b03484926a0c18c6"
format_stamp: "Formatted at 2026-09-29 02:08:59 on dist-test-slave-q06p"
server_key: "94e42ec9a2170929b90d614ebe83e936"
server_key_iv: "098f1dc345e838f2deceabc1470c7574"
server_key_version: "kuduclusterkey@0"
I20260929 02:08:59.668135 24421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:08:59.694528 24421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:08:59.699874 24421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:08:59.700276 24421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:08:59.701515 24421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:08:59.704403 24421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:08:59.704547 24421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:08:59.704751 24421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:08:59.704864 24421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:08:59 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647739, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:08:59.893421 24421 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:45987
I20260929 02:08:59.893462 24553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:45987 every 8 connection(s)
I20260929 02:08:59.895226 24421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:08:59.899459 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 24421
I20260929 02:08:59.899672 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:08:59.900463 11995 external_mini_cluster.cc:1428] Setting key bece04e3883d230393274b6494a9c31c
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
Sep 29 02:08:59 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647739, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:08:59.944154 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:59.901239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:52067 (local address 127.11.182.254:33067)
0929 02:08:59.923335 (+ 22096us) server_negotiation.cc:206] Beginning negotiation
0929 02:08:59.923345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:08:59.924590 (+ 1245us) server_negotiation.cc:407] Connection header received
0929 02:08:59.927062 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:08:59.927067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:08:59.927178 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:08:59.927302 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:08:59.929292 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:59.930059 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:59.932377 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:08:59.932789 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:08:59.932925 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:59.937134 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:08:59.937139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:08:59.937142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:08:59.937173 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:08:59.940533 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:59.941403 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:59.941408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:59.941412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:59.941576 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:08:59.942088 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:08:59.942094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:08:59.942098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:08:59.942434 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:08:59.942570 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:08:59.943554 (+ 984us) server_negotiation.cc:299] Negotiation successful
0929 02:08:59.943775 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21912,"thread_start_us":21786,"threads_started":1}
I20260929 02:08:59.946528 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:08:59.898559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33067 (local address 127.11.182.193:52067)
0929 02:08:59.923996 (+ 25437us) negotiation.cc:107] Waiting for socket to connect
0929 02:08:59.924052 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:08:59.926704 (+ 2652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:08:59.927609 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:08:59.927624 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:08:59.928283 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:08:59.929123 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:59.929151 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:59.930219 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:59.930224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:59.932226 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:08:59.932241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:08:59.933028 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:08:59.933037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:08:59.933322 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:08:59.934155 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:08:59.934184 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:08:59.936901 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:08:59.940692 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:08:59.940702 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:08:59.940721 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:08:59.941268 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:08:59.941720 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:08:59.941724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:08:59.941728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:08:59.941961 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:08:59.942667 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:08:59.942675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:08:59.943111 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 02:08:59.944687 (+ 1576us) client_negotiation.cc:240] Negotiation successful
0929 02:08:59.945213 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25123,"thread_start_us":24898,"threads_started":1}
I20260929 02:08:59.948602 24554 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33067
I20260929 02:08:59.948905 24554 heartbeater.cc:464] Registering TS with master...
I20260929 02:08:59.949587 24554 heartbeater.cc:511] Master 127.11.182.254:33067 requested a full tablet report, sending...
I20260929 02:08:59.952296 24340 ts_manager.cc:194] Registered new tserver with Master: cedd592820f84c90b03484926a0c18c6 (127.11.182.193:45987)
I20260929 02:08:59.954330 24340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:52067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:08:59.970129 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45647/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33067
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:09:00 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647739, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
W20260929 02:09:00.405004 24560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:00.405309 24560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:00.405479 24560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:00.412926 24560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:00.413067 24560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:00.413230 24560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:09:00.423024 24560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.182.212:45647/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:00.424329 24560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:00.425608 24560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:00.437904 24574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:00.438038 24575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:00.438426 24577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:00.439083 24560 server_base.cc:1034] running on GCE node
I20260929 02:09:00.439618 24560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:00.440614 24560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:00.441834 24560 hybrid_clock.cc:648] HybridClock initialized: now 1790647740441793 us; error 62 us; skew 500 ppm
I20260929 02:09:00.442137 24560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:00 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647740, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:00.445200 24560 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:09:00.446645 24560 webserver.cc:460] Webserver started at http://127.11.182.194:44517/ using document root <none> and password file <none>
I20260929 02:09:00.447309 24560 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:00.447404 24560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:00.447657 24560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:09:00 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647740, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:09:00.462648 24560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "2d9778426f6443119cf57a0720f57ba7"
format_stamp: "Formatted at 2026-09-29 02:09:00 on dist-test-slave-q06p"
server_key: "b756c92be36fa86db9499fee5e2fe3e9"
server_key_iv: "12e835bdea2c46d698f58f4cc9aa84e2"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:00.463467 24560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "2d9778426f6443119cf57a0720f57ba7"
format_stamp: "Formatted at 2026-09-29 02:09:00 on dist-test-slave-q06p"
server_key: "b756c92be36fa86db9499fee5e2fe3e9"
server_key_iv: "12e835bdea2c46d698f58f4cc9aa84e2"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:00.468465 24560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:09:00.484388 24584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:00.485986 24560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:09:00.486176 24560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "2d9778426f6443119cf57a0720f57ba7"
format_stamp: "Formatted at 2026-09-29 02:09:00 on dist-test-slave-q06p"
server_key: "b756c92be36fa86db9499fee5e2fe3e9"
server_key_iv: "12e835bdea2c46d698f58f4cc9aa84e2"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:00.486409 24560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:00.500334 24560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:00.503799 24560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:00.504089 24560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:00.504964 24560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:00.506243 24560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:00.506345 24560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:00.506500 24560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:00.506577 24560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:00.538156 24560 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:40145
I20260929 02:09:00.538190 24696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:40145 every 8 connection(s)
I20260929 02:09:00.539526 24560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:09:00.542057 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 24560
I20260929 02:09:00.542232 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:09:00.542781 11995 external_mini_cluster.cc:1428] Setting key 9d7ce301c94582479363b5c47405c9c3
Sep 29 02:09:00 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647740, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:00.561249 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:00.541950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:34963 (local address 127.11.182.254:33067)
0929 02:09:00.542428 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:00.542432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:00.543597 (+ 1165us) server_negotiation.cc:407] Connection header received
0929 02:09:00.545748 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:00.545756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:00.545854 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:00.545968 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:00.547354 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:00.548547 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:00.550005 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:00.550730 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:00.550894 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:00.554255 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:00.554264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:00.554272 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:00.554315 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:00.557909 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:00.558600 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:00.558609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:00.558616 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:00.558800 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:00.559283 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:00.559291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:00.559297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:00.559671 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:00.559835 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:00.560483 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 02:09:00.560767 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":254,"threads_started":1}
I20260929 02:09:00.561851 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:00.542613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33067 (local address 127.11.182.194:34963)
0929 02:09:00.543421 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:00.543469 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:00.545498 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:00.546158 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:00.546165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:00.546563 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:00.547188 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:00.547213 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:00.548684 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:00.548688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:00.549868 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:00.549887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:00.550966 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:00.550972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:00.551170 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:00.551900 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:00.551931 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:00.554017 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:00.558035 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:00.558042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:00.558054 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:00.558467 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:00.558948 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:00.558951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:00.558953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:00.559155 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:00.559794 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:00.559800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:00.560090 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 02:09:00.560325 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 02:09:00.560753 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":362,"threads_started":1}
I20260929 02:09:00.563344 24697 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33067
I20260929 02:09:00.563680 24697 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:00.564584 24697 heartbeater.cc:511] Master 127.11.182.254:33067 requested a full tablet report, sending...
I20260929 02:09:00.566212 24340 ts_manager.cc:194] Registered new tserver with Master: 2d9778426f6443119cf57a0720f57ba7 (127.11.182.194:40145)
I20260929 02:09:00.567077 24340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:34963
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:09:00.598389 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45647/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33067
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:00.912819 24703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:00.913149 24703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:00.913327 24703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:00.920552 24703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:00.920679 24703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:00.920855 24703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:09:00.930459 24703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.182.212:45647/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:00.931713 24703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:00.932904 24703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:00.945713 24708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:00.945749 24709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:00.945786 24711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:00.946409 24703 server_base.cc:1034] running on GCE node
I20260929 02:09:00.947036 24703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:00.947933 24703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:00.949118 24703 hybrid_clock.cc:648] HybridClock initialized: now 1790647740949059 us; error 58 us; skew 500 ppm
I20260929 02:09:00.949406 24703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:00 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647740, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:00.952466 24703 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:09:00.953938 24703 webserver.cc:460] Webserver started at http://127.11.182.195:43405/ using document root <none> and password file <none>
I20260929 02:09:00.954579 24703 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:00.954720 24703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:00.955077 24703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:09:00 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647740, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:09:00.957954 24554 heartbeater.cc:503] Master 127.11.182.254:33067 was elected leader, sending a full tablet report...
I20260929 02:09:00.968811 24703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "2d374d98d12b42d6a9c4f1bfcf717557"
format_stamp: "Formatted at 2026-09-29 02:09:00 on dist-test-slave-q06p"
server_key: "620968741889555ca3275c2d629e473f"
server_key_iv: "f18949a39e0ec35fb44b18ee95b6ccc5"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:00.969575 24703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "2d374d98d12b42d6a9c4f1bfcf717557"
format_stamp: "Formatted at 2026-09-29 02:09:00 on dist-test-slave-q06p"
server_key: "620968741889555ca3275c2d629e473f"
server_key_iv: "f18949a39e0ec35fb44b18ee95b6ccc5"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:00.974954 24703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:09:00.991164 24718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:00.992823 24703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:09:00.992996 24703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "2d374d98d12b42d6a9c4f1bfcf717557"
format_stamp: "Formatted at 2026-09-29 02:09:00 on dist-test-slave-q06p"
server_key: "620968741889555ca3275c2d629e473f"
server_key_iv: "f18949a39e0ec35fb44b18ee95b6ccc5"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:00.993259 24703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:01.021674 24703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:01.025430 24703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:01.025676 24703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:01.026502 24703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:01.027935 24703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:01.028046 24703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:01.028141 24703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:01.028219 24703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:01.062708 24703 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:45079
I20260929 02:09:01.062731 24830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:45079 every 8 connection(s)
I20260929 02:09:01.063934 24703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:09:01.069720 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 24703
I20260929 02:09:01.069901 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:09:01.070487 11995 external_mini_cluster.cc:1428] Setting key 4823425e32a37f76890d760748b46d15
Sep 29 02:09:01 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647740, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:01.082484 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:01.066364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:39037 (local address 127.11.182.254:33067)
0929 02:09:01.066864 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:01.066872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:01.067721 (+ 849us) server_negotiation.cc:407] Connection header received
0929 02:09:01.069368 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:01.069373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:01.069448 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:01.069542 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:01.071335 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.072251 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.073943 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.074371 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.074462 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:01.077565 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:01.077570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:01.077574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:01.077602 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:01.080098 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:01.080723 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:01.080727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:01.080729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:01.080841 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:01.081204 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:01.081207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:01.081209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:01.081440 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:01.081542 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:01.082097 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 02:09:01.082243 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":216,"threads_started":1}
I20260929 02:09:01.083418 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:01.066898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33067 (local address 127.11.182.195:39037)
0929 02:09:01.067575 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:01.067609 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:01.069118 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:01.069806 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:01.069818 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:01.070386 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:01.071169 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:01.071188 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.072413 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.072428 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:01.073777 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:01.073794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.074558 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.074563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:01.074750 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:01.075394 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:01.075415 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:01.077382 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:01.080223 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:01.080229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:01.080239 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:01.080610 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:01.080926 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:01.080929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:01.080931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:01.081102 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:01.081535 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:01.081540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:01.081796 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 02:09:01.082004 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 02:09:01.082355 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":262,"threads_started":1}
I20260929 02:09:01.084867 24831 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33067
I20260929 02:09:01.085177 24831 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:01.085911 24831 heartbeater.cc:511] Master 127.11.182.254:33067 requested a full tablet report, sending...
I20260929 02:09:01.087247 24339 ts_manager.cc:194] Registered new tserver with Master: 2d374d98d12b42d6a9c4f1bfcf717557 (127.11.182.195:45079)
I20260929 02:09:01.088048 24339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:39037
I20260929 02:09:01.097801 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:09:01.099438 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 24421
Sep 29 02:09:01 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:09:01.114737 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:01.100241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.11.182.193:45987)
0929 02:09:01.100811 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:01.100817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:01.100839 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:09:01.100894 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:01.100900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:01.101061 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:01.101183 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:01.102466 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.103535 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.105067 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.105581 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.105670 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:01.109636 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:01.109646 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:01.109659 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:01.109691 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:01.112471 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:01.113058 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:01.113064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:01.113067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:01.113212 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:01.113535 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:01.113538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:01.113540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:01.113831 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:01.114046 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:01.114293 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 02:09:01.114476 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":292,"threads_started":1}
I20260929 02:09:01.115944 24549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:01.204562 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 24421
I20260929 02:09:01.220289 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 24560
Sep 29 02:09:01 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:09:01.237108 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:01.221190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.11.182.194:40145)
0929 02:09:01.221745 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:01.221772 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:01.221804 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:09:01.221890 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:01.221900 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:01.222229 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:01.222408 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:01.223753 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.224740 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.226112 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.226672 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.226805 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:01.231648 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:01.231661 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:01.231686 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:01.231725 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:01.234256 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:01.234923 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:01.234933 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:01.234937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:01.235084 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:01.235476 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:01.235481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:01.235484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:01.235819 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:01.236129 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:01.236480 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 02:09:01.236742 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":294,"threads_started":1}
I20260929 02:09:01.238356 24692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:01.319476 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 24560
I20260929 02:09:01.339134 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 24703
Sep 29 02:09:01 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:09:01.355099 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:01.340229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.11.182.195:45079)
0929 02:09:01.340841 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:01.340848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:01.340876 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:09:01.340940 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:01.340946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:01.341164 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:01.341342 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:01.342734 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.343872 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.345160 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:01.345704 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:01.345836 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:01.350178 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:01.350188 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:01.350203 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:01.350237 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:01.352749 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:01.353351 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:01.353357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:01.353359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:01.353477 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:01.353770 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:01.353773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:01.353775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:01.354071 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:01.354300 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:01.354523 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 02:09:01.354705 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":316,"threads_started":1}
I20260929 02:09:01.356328 24826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:01.437812 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 24703
I20260929 02:09:01.453469 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 24308
I20260929 02:09:01.454648 24370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:01.559754 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 24308
I20260929 02:09:01.580948 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33067
--webserver_interface=127.11.182.254
--webserver_port=34241
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:33067
--ranger_config_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45647/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.182.212:45647/kms with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:01.896514 24843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:01.896797 24843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:01.896884 24843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:01.904168 24843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:09:01.904287 24843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:01.904376 24843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:01.904474 24843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:09:01.904557 24843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:09:01.914353 24843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.182.212:45647/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:33067
--ranger_config_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=34241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:01.915565 24843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:01.916836 24843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:01.928488 24848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:01.928449 24851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:01.928531 24849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:01.932390 24843 server_base.cc:1034] running on GCE node
I20260929 02:09:01.933099 24843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:01.934424 24843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:01.935647 24843 hybrid_clock.cc:648] HybridClock initialized: now 1790647741935607 us; error 52 us; skew 500 ppm
I20260929 02:09:01.935976 24843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:01 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647741, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:01.938930 24843 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:09:01.940392 24843 webserver.cc:460] Webserver started at http://127.11.182.254:34241/ using document root <none> and password file <none>
I20260929 02:09:01.940973 24843 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:01.941075 24843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:09:01 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647741, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:09:02.011247 24843 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:09:02.014001 24859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:02.015399 24843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:09:02.015550 24843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "788c52b5896648f7be8f55d8dc9f6b11"
format_stamp: "Formatted at 2026-09-29 02:08:58 on dist-test-slave-q06p"
server_key: "63655c8ed2b144b97bc0dd61a96218ff"
server_key_iv: "fe93f040466a124a71324be60546bfe5"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:02.016444 24843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:02.041940 24843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:02.046015 24843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:02.046250 24843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:02.064899 24843 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:33067
I20260929 02:09:02.064919 24910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:33067 every 8 connection(s)
I20260929 02:09:02.066277 24843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:09:02.072341 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 24843
I20260929 02:09:02.072786 24911 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:09:02.073226 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45647/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:45987
--local_ip_for_outbound_sockets=127.11.182.193
--tserver_master_addrs=127.11.182.254:33067
--webserver_port=35741
--webserver_interface=127.11.182.193
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.182.212:45647/kms with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:09:02.076056 24911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Bootstrap starting.
I20260929 02:09:02.093920 24911 log.cc:824] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:02.102399 24911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:09:02.102977 24911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Bootstrap complete.
I20260929 02:09:02.111653 24911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } }
I20260929 02:09:02.112465 24911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788c52b5896648f7be8f55d8dc9f6b11, State: Initialized, Role: FOLLOWER
I20260929 02:09:02.113013 24911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } }
I20260929 02:09:02.113246 24911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:09:02.113369 24911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:09:02.113543 24911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:09:02.117274 24911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } }
I20260929 02:09:02.117735 24911 leader_election.cc:302] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788c52b5896648f7be8f55d8dc9f6b11; no voters:
I20260929 02:09:02.118358 24911 leader_election.cc:288] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:09:02.118485 24915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:09:02.119765 24915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [term 2 LEADER]: Becoming Leader. State: Replica: 788c52b5896648f7be8f55d8dc9f6b11, State: Running, Role: LEADER
I20260929 02:09:02.120368 24915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } }
I20260929 02:09:02.120576 24911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:09:02.123013 24917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788c52b5896648f7be8f55d8dc9f6b11. Latest consensus state: current_term: 2 leader_uuid: "788c52b5896648f7be8f55d8dc9f6b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } } }
I20260929 02:09:02.123368 24917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:02.123070 24916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "788c52b5896648f7be8f55d8dc9f6b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c52b5896648f7be8f55d8dc9f6b11" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33067 } } }
I20260929 02:09:02.123490 24916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:02.137945 24921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:09:02.143503 24921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:09:02.144322 24911 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:09:02.145097 24921 catalog_manager.cc:1259] Loaded cluster ID: 2e769a7298914e19bb89d9d497f7b763
I20260929 02:09:02.145274 24921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:09:02.149302 24921 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:09:02.150738 24921 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Loaded TSK: 0
I20260929 02:09:02.152336 24921 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:09:02.509439 24913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:02.509989 24913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:02.510198 24913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:02.521768 24913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:02.521981 24913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:02.522228 24913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:09:02.537194 24913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.182.212:45647/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:45987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=35741
--tserver_master_addrs=127.11.182.254:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:02.539326 24913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:02.540845 24913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:02.576434 24949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:02.576704 24951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:02.577175 24948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:02.578922 24913 server_base.cc:1034] running on GCE node
I20260929 02:09:02.579879 24913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:02.580991 24913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:02.582211 24913 hybrid_clock.cc:648] HybridClock initialized: now 1790647742582139 us; error 62 us; skew 500 ppm
I20260929 02:09:02.582553 24913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:02 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647742, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:02.586527 24913 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:09:02.591886 24913 webserver.cc:460] Webserver started at http://127.11.182.193:35741/ using document root <none> and password file <none>
I20260929 02:09:02.592617 24913 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:02.592783 24913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:09:02 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647742, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:09:02.622680 24913 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.005s sys 0.004s
I20260929 02:09:02.626114 24959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:02.628082 24913 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:09:02.628297 24913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "cedd592820f84c90b03484926a0c18c6"
format_stamp: "Formatted at 2026-09-29 02:08:59 on dist-test-slave-q06p"
server_key: "94e42ec9a2170929b90d614ebe83e936"
server_key_iv: "098f1dc345e838f2deceabc1470c7574"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:02.629300 24913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:02.705706 24913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:02.711195 24913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:02.711575 24913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:02.717870 24913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:02.719521 24913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:02.719643 24913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:02.719751 24913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:02.719826 24913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:02.793567 24913 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:45987
I20260929 02:09:02.793641 25072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:45987 every 8 connection(s)
I20260929 02:09:02.795146 24913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:09:02.797003 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 24913
I20260929 02:09:02.797714 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45647/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:40145
--local_ip_for_outbound_sockets=127.11.182.194
--tserver_master_addrs=127.11.182.254:33067
--webserver_port=44517
--webserver_interface=127.11.182.194
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.182.212:45647/kms with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:09:02 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647742, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:02.831240 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:02.802324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:53911 (local address 127.11.182.254:33067)
0929 02:09:02.803181 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:02.803200 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:02.803252 (+ 52us) server_negotiation.cc:407] Connection header received
0929 02:09:02.805214 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:02.805236 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:02.805744 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:02.806068 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:02.809107 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:02.811317 (+ 2210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:02.813945 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:02.815101 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:02.815291 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:02.820805 (+ 5514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:02.820821 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:02.820847 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:02.820907 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:02.825474 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:02.826657 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:02.826668 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:02.826678 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:02.826927 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:02.827512 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:02.827517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:02.827521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:02.828170 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:02.828346 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:02.829460 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0929 02:09:02.829881 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":1031,"threads_started":1}
I20260929 02:09:02.831243 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:02.801129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33067 (local address 127.11.182.193:53911)
0929 02:09:02.801957 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:02.802007 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:02.804433 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:02.807395 (+ 2962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:02.807410 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:02.808071 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:02.808863 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:02.808890 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:02.811525 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:02.811544 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:02.813761 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:02.813780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:02.815737 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:02.815746 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:02.816079 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:02.816933 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:02.816972 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:02.820306 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:02.825707 (+ 5401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:02.825720 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:02.825742 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:02.826437 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:02.827086 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:02.827092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:02.827096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:02.827365 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:02.828368 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:02.828380 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:02.828853 (+ 473us) client_negotiation.cc:769] Sending connection context
0929 02:09:02.829221 (+ 368us) client_negotiation.cc:240] Negotiation successful
0929 02:09:02.829825 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":1368,"threads_started":1}
I20260929 02:09:02.835659 25073 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33067
I20260929 02:09:02.835974 25073 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:02.836792 25073 heartbeater.cc:511] Master 127.11.182.254:33067 requested a full tablet report, sending...
W20260929 02:09:02.839942 25073 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:33067 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:09:02.840188 25073 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:02.840471 25073 heartbeater.cc:511] Master 127.11.182.254:33067 requested a full tablet report, sending...
I20260929 02:09:02.841912 25073 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:02.842255 25073 heartbeater.cc:511] Master 127.11.182.254:33067 requested a full tablet report, sending...
W20260929 02:09:02.843426 25073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:09:03.190095 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:03.190546 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:03.190728 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:03.201135 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:03.201269 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:03.201493 25077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:09:03.215776 25077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.182.212:45647/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:40145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=44517
--tserver_master_addrs=127.11.182.254:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:03.217391 25077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:03.219099 25077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:03.236428 25091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:03.236399 25092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:03.237350 25094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:03.237637 25077 server_base.cc:1034] running on GCE node
I20260929 02:09:03.238351 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:03.239519 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:03.240830 25077 hybrid_clock.cc:648] HybridClock initialized: now 1790647743240720 us; error 100 us; skew 500 ppm
I20260929 02:09:03.241199 25077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:03 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647743, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:03.244390 25077 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:09:03.276203 25077 webserver.cc:460] Webserver started at http://127.11.182.194:44517/ using document root <none> and password file <none>
I20260929 02:09:03.277305 25077 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:03.277568 25077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:09:03 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647743, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:09:03.317202 25077 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:09:03.321033 25104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:03.322896 25077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:09:03.323071 25077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "2d9778426f6443119cf57a0720f57ba7"
format_stamp: "Formatted at 2026-09-29 02:09:00 on dist-test-slave-q06p"
server_key: "b756c92be36fa86db9499fee5e2fe3e9"
server_key_iv: "12e835bdea2c46d698f58f4cc9aa84e2"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:03.324506 25077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 02:09:03 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647743, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:03.372651 25077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:03.378168 25077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:03.378573 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:03.386171 25077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:03.388713 25077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:03.388892 25077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 02:09:03.389060 25077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:03.389179 25077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:03.437985 25077 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:40145
I20260929 02:09:03.438017 25216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:40145 every 8 connection(s)
I20260929 02:09:03.440054 25077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:09:03.450096 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 25077
I20260929 02:09:03.450888 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.182.212:45647/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:45079
--local_ip_for_outbound_sockets=127.11.182.195
--tserver_master_addrs=127.11.182.254:33067
--webserver_port=43405
--webserver_interface=127.11.182.195
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.182.212:45647/kms with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:09:03 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647743, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:03.468317 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:03.443327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33067 (local address 127.11.182.194:52163)
0929 02:09:03.444100 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:03.444152 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:03.445852 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:03.446623 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:03.446634 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:03.447277 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:03.448111 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:03.448138 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:03.450154 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:03.450160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:03.452598 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:03.452628 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:03.454060 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:03.454070 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:03.454407 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:03.455315 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:03.455358 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:03.458337 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:03.463016 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:03.463029 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:03.463050 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:03.463692 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:03.464216 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:03.464224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:03.464231 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:03.464504 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:03.465287 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:03.465297 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:03.465756 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 02:09:03.466048 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 02:09:03.466654 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":330,"threads_started":1}
I20260929 02:09:03.469857 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:03.442723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:52163 (local address 127.11.182.254:33067)
0929 02:09:03.443222 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:03.443229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:03.444259 (+ 1030us) server_negotiation.cc:407] Connection header received
0929 02:09:03.446140 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:03.446146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:03.446241 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:03.446360 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:03.448321 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:03.449522 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:03.452801 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:03.453542 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:03.453709 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:03.458653 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:03.458661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:03.458666 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:03.458713 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:03.462798 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:03.463892 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:03.463900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:03.463906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:03.464060 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:03.464644 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:03.464652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:03.464658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:03.465042 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:03.465213 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:03.469139 (+ 3926us) server_negotiation.cc:299] Negotiation successful
0929 02:09:03.469398 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":251,"threads_started":1}
I20260929 02:09:03.472348 25217 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33067
I20260929 02:09:03.472682 25217 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:03.473661 25217 heartbeater.cc:511] Master 127.11.182.254:33067 requested a full tablet report, sending...
I20260929 02:09:03.476117 24876 ts_manager.cc:194] Registered new tserver with Master: 2d9778426f6443119cf57a0720f57ba7 (127.11.182.194:40145)
I20260929 02:09:03.478574 24876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:52163
Sep 29 02:09:03 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647743, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:09:03.843550 25073 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:03.843961 25073 heartbeater.cc:511] Master 127.11.182.254:33067 requested a full tablet report, sending...
I20260929 02:09:03.845263 24876 ts_manager.cc:194] Registered new tserver with Master: cedd592820f84c90b03484926a0c18c6 (127.11.182.193:45987)
I20260929 02:09:03.846547 24876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:53911
W20260929 02:09:03.905707 25222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:03.906103 25222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:03.906231 25222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:03.916298 25222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:03.916435 25222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:03.916584 25222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:09:03.928529 25222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.182.212:45647/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:45079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=43405
--tserver_master_addrs=127.11.182.254:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:03.929940 25222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:03.931407 25222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:03.946694 25228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:03.947531 25227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:03.951973 25230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:03.952843 25222 server_base.cc:1034] running on GCE node
I20260929 02:09:03.953538 25222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:03.957355 25222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:03.958606 25222 hybrid_clock.cc:648] HybridClock initialized: now 1790647743958544 us; error 55 us; skew 500 ppm
I20260929 02:09:03.958988 25222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:03 dist-test-slave-q06p krb5kdc[23864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647743, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:03.962277 25222 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:09:03.967235 25222 webserver.cc:460] Webserver started at http://127.11.182.195:43405/ using document root <none> and password file <none>
I20260929 02:09:03.967958 25222 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:03.968122 25222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:09:03 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647743, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:09:04.019373 25222 fs_manager.cc:709] Time spent opening directory manager: real 0.037s user 0.025s sys 0.012s
I20260929 02:09:04.025146 25238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:04.026985 25222 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s
I20260929 02:09:04.027182 25222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "2d374d98d12b42d6a9c4f1bfcf717557"
format_stamp: "Formatted at 2026-09-29 02:09:00 on dist-test-slave-q06p"
server_key: "620968741889555ca3275c2d629e473f"
server_key_iv: "f18949a39e0ec35fb44b18ee95b6ccc5"
server_key_version: "kuduclusterkey@0"
I20260929 02:09:04.028218 25222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:04.058655 25222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:04.064637 25222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:04.064973 25222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:04.075311 25222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:04.080196 25222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:04.080327 25222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:04.080474 25222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:04.080579 25222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:09:04 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647743, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for HTTP/127.11.182.212@KRBTEST.COM
I20260929 02:09:04.323910 25222 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:45079
I20260929 02:09:04.323993 25351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:45079 every 8 connection(s)
I20260929 02:09:04.325809 25222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:09:04.334896 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 25222
Sep 29 02:09:04 dist-test-slave-q06p krb5kdc[23864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647743, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:04.403997 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.329904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:55531 (local address 127.11.182.254:33067)
0929 02:09:04.371825 (+ 41921us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.371836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.374905 (+ 3069us) server_negotiation.cc:407] Connection header received
0929 02:09:04.382971 (+ 8066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.382978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.383051 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.383130 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:04.387232 (+ 4102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.388426 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.390945 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.391561 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.391659 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.396869 (+ 5210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:04.396879 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:04.396887 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:04.396931 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:04.400300 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.401217 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.401224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.401232 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.401377 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.401964 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.401969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.401973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.402323 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:04.402476 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.403491 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.403709 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41703,"thread_start_us":48944,"threads_started":1}
I20260929 02:09:04.405244 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.336328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.11.182.254:33067)
0929 02:09:04.372315 (+ 35987us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.372322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.381479 (+ 9157us) server_negotiation.cc:407] Connection header received
0929 02:09:04.382071 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.382078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.382179 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.382267 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:04.385494 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.386676 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.392950 (+ 6274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.393626 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.393742 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.398371 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:04.398378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:04.398383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:04.398439 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:04.401842 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.402513 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.402522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.402524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.402645 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.403094 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.403099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.403104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.403369 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:04.403631 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.404689 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.404889 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35864,"thread_start_us":35523,"threads_started":1}
I20260929 02:09:04.405471 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.329309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:33067 (local address 127.11.182.195:55531)
0929 02:09:04.373012 (+ 43703us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.373079 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:04.380825 (+ 7746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:04.383837 (+ 3012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:04.383854 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:04.384623 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:04.386996 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.387025 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.388659 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.388677 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.390727 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.390769 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.392043 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.392052 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.392332 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.393454 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:04.393485 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:04.396575 (+ 3090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:04.400446 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:04.400457 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:04.400475 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:04.401064 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:04.401523 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:04.401530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:04.401537 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:04.401808 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:04.402571 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:04.402580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:04.403062 (+ 482us) client_negotiation.cc:769] Sending connection context
0929 02:09:04.403381 (+ 319us) client_negotiation.cc:240] Negotiation successful
0929 02:09:04.403948 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43307,"thread_start_us":49441,"threads_started":1}
I20260929 02:09:04.407110 25352 heartbeater.cc:347] Connected to a master server at 127.11.182.254:33067
I20260929 02:09:04.407426 25352 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:04.408277 25352 heartbeater.cc:511] Master 127.11.182.254:33067 requested a full tablet report, sending...
I20260929 02:09:04.409647 24876 ts_manager.cc:194] Registered new tserver with Master: 2d374d98d12b42d6a9c4f1bfcf717557 (127.11.182.195:45079)
I20260929 02:09:04.410431 24876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:55531
I20260929 02:09:04.421139 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:09:04.441159 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.424849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.11.182.254:33067)
0929 02:09:04.424997 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.425001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.425401 (+ 400us) server_negotiation.cc:407] Connection header received
0929 02:09:04.425552 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.425556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.425639 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.425714 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:04.427769 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.428474 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.430483 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.431025 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.431144 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.434460 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:04.434464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:04.434467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:04.434498 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:04.437507 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.438279 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.438284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.438288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.438431 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.438907 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.438912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.438916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.439233 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:04.439463 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.440634 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.440853 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 02:09:04.446519 24876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:09:04.453992 24876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:09:04.482264 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.468949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.11.182.195:45079)
0929 02:09:04.469577 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.469585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.470465 (+ 880us) server_negotiation.cc:407] Connection header received
0929 02:09:04.470936 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.470948 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.471262 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.471503 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:04.472815 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.474887 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.478094 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.480157 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.480400 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.480627 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.481514 (+ 887us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.481756 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":339,"threads_started":1}
I20260929 02:09:04.483481 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.468327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45079 (local address 127.0.0.1:60204)
0929 02:09:04.470291 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.470341 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:04.470614 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:04.471989 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:04.471999 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:04.472039 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:04.472568 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.472583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.475058 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.475062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.477886 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.477905 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.480881 (+ 2976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.480890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.481189 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.481261 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 02:09:04.482879 (+ 1618us) client_negotiation.cc:240] Negotiation successful
0929 02:09:04.483096 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1726,"threads_started":1}
I20260929 02:09:04.484074 25217 heartbeater.cc:503] Master 127.11.182.254:33067 was elected leader, sending a full tablet report...
I20260929 02:09:04.488386 25287 tablet_service.cc:1467] Processing CreateTablet for tablet f6809945a976423aac08caa52e5d762e (DEFAULT_TABLE table=test-table [id=d86873a612f24fbb9400a7faf8d0ec0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:04.490406 25287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6809945a976423aac08caa52e5d762e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:04.496745 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.469572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45987 (local address 127.0.0.1:47220)
0929 02:09:04.483071 (+ 13499us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.483095 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:04.483207 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:04.485508 (+ 2301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:04.485516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:04.485551 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:04.485957 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.485971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.488706 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.488714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.491417 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.491431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.494194 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.494203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.494450 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.494504 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 02:09:04.494624 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 02:09:04.494824 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13383,"thread_start_us":367,"threads_started":1}
I20260929 02:09:04.497905 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.468973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.11.182.193:45987)
0929 02:09:04.483765 (+ 14792us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.484533 (+ 768us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.484569 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:09:04.484686 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.484694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.484974 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.485202 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:04.486199 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.488482 (+ 2283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.491602 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.493554 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.493807 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.494019 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.497406 (+ 3387us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.497589 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14607,"thread_start_us":14433,"threads_started":1}
I20260929 02:09:04.502318 25008 tablet_service.cc:1467] Processing CreateTablet for tablet f6809945a976423aac08caa52e5d762e (DEFAULT_TABLE table=test-table [id=d86873a612f24fbb9400a7faf8d0ec0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:04.503316 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.484091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:40145 (local address 127.0.0.1:52766)
0929 02:09:04.491234 (+ 7143us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.491257 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:04.491375 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:04.496398 (+ 5023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:04.496403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:04.496431 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:04.496603 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.496612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.498669 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.498674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.500756 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.500771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.502574 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.502580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.502761 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.502803 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:09:04.502899 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:09:04.503057 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7004}
I20260929 02:09:04.504590 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.484413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.11.182.194:40145)
0929 02:09:04.495535 (+ 11122us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.495545 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.495578 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:09:04.495681 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.495691 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.495965 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.496776 (+ 811us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:04.497034 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.498517 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.500964 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.502377 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.503841 (+ 1464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.504021 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.504094 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.504250 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10936,"thread_start_us":11712,"threads_started":1}
I20260929 02:09:04.504814 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6809945a976423aac08caa52e5d762e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:04.507953 25152 tablet_service.cc:1467] Processing CreateTablet for tablet f6809945a976423aac08caa52e5d762e (DEFAULT_TABLE table=test-table [id=d86873a612f24fbb9400a7faf8d0ec0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:04.510526 25152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6809945a976423aac08caa52e5d762e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:04.527655 25379 tablet_bootstrap.cc:492] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: Bootstrap starting.
I20260929 02:09:04.530544 25380 tablet_bootstrap.cc:492] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: Bootstrap starting.
I20260929 02:09:04.531740 25381 tablet_bootstrap.cc:492] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: Bootstrap starting.
I20260929 02:09:04.531934 25379 tablet_bootstrap.cc:654] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:04.533444 25379 log.cc:824] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:04.534389 25380 tablet_bootstrap.cc:654] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:04.536808 25381 tablet_bootstrap.cc:654] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:04.536881 25380 log.cc:824] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:04.538125 25381 log.cc:824] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:04.542968 25379 tablet_bootstrap.cc:492] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: No bootstrap required, opened a new log
I20260929 02:09:04.543286 25379 ts_tablet_manager.cc:1397] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260929 02:09:04.546653 25380 tablet_bootstrap.cc:492] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: No bootstrap required, opened a new log
I20260929 02:09:04.547034 25380 ts_tablet_manager.cc:1397] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260929 02:09:04.554924 25379 raft_consensus.cc:348] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } }
I20260929 02:09:04.555645 25379 raft_consensus.cc:374] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:04.555785 25379 raft_consensus.cc:729] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d374d98d12b42d6a9c4f1bfcf717557, State: Initialized, Role: FOLLOWER
I20260929 02:09:04.556598 25379 consensus_queue.cc:260] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } }
I20260929 02:09:04.560149 25352 heartbeater.cc:503] Master 127.11.182.254:33067 was elected leader, sending a full tablet report...
I20260929 02:09:04.560774 25379 ts_tablet_manager.cc:1428] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260929 02:09:04.562992 25381 tablet_bootstrap.cc:492] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: No bootstrap required, opened a new log
I20260929 02:09:04.563333 25381 ts_tablet_manager.cc:1397] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.008s
I20260929 02:09:04.563963 25380 raft_consensus.cc:348] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } }
I20260929 02:09:04.564447 25380 raft_consensus.cc:374] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:04.564590 25380 raft_consensus.cc:729] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedd592820f84c90b03484926a0c18c6, State: Initialized, Role: FOLLOWER
I20260929 02:09:04.565343 25380 consensus_queue.cc:260] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } }
I20260929 02:09:04.568361 25073 heartbeater.cc:503] Master 127.11.182.254:33067 was elected leader, sending a full tablet report...
I20260929 02:09:04.569063 25380 ts_tablet_manager.cc:1428] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: Time spent starting tablet: real 0.022s user 0.017s sys 0.000s
I20260929 02:09:04.572817 25381 raft_consensus.cc:348] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } }
I20260929 02:09:04.573289 25381 raft_consensus.cc:374] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:04.573382 25381 raft_consensus.cc:729] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9778426f6443119cf57a0720f57ba7, State: Initialized, Role: FOLLOWER
I20260929 02:09:04.573961 25381 consensus_queue.cc:260] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } }
I20260929 02:09:04.575593 25381 ts_tablet_manager.cc:1428] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260929 02:09:04.604637 25386 raft_consensus.cc:482] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:09:04.604965 25386 raft_consensus.cc:504] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } }
I20260929 02:09:04.606590 25386 leader_election.cc:288] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d374d98d12b42d6a9c4f1bfcf717557 (127.11.182.195:45079), 2d9778426f6443119cf57a0720f57ba7 (127.11.182.194:40145)
I20260929 02:09:04.614614 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.608073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:43641 (local address 127.11.182.194:40145)
0929 02:09:04.608266 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.608273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.608639 (+ 366us) server_negotiation.cc:407] Connection header received
0929 02:09:04.608860 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.608864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.608919 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.609024 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:04.609578 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.610491 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.612450 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.613678 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.613851 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.613926 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.614237 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.614372 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 02:09:04.614645 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.607849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:40145 (local address 127.11.182.193:43641)
0929 02:09:04.608524 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.608541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:04.608715 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:04.609036 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:04.609040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:04.609060 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:04.609424 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.609434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.610659 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.610664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.612293 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.612305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.613891 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.613897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.614058 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.614094 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:09:04.614194 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:09:04.614323 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":427,"threads_started":1}
I20260929 02:09:04.614837 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.607006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:51867 (local address 127.11.182.195:45079)
0929 02:09:04.607242 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.607251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.607877 (+ 626us) server_negotiation.cc:407] Connection header received
0929 02:09:04.608157 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.608163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.608249 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.608310 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:04.609085 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.610134 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.612615 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.613777 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.613902 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.613977 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.614454 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.614575 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:09:04.615487 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.607007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45079 (local address 127.11.182.193:51867)
0929 02:09:04.607769 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.607790 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:04.607999 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:04.608504 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:04.608508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:04.608539 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:04.608913 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.608925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.610317 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.610322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.612462 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:04.612476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.614094 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.614099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:04.614278 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.614309 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:09:04.615012 (+ 703us) client_negotiation.cc:240] Negotiation successful
0929 02:09:04.615154 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"spinlock_wait_cycles":5888,"thread_start_us":396,"threads_started":1}
I20260929 02:09:04.616053 25172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6809945a976423aac08caa52e5d762e" candidate_uuid: "cedd592820f84c90b03484926a0c18c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9778426f6443119cf57a0720f57ba7" is_pre_election: true
I20260929 02:09:04.616376 25307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6809945a976423aac08caa52e5d762e" candidate_uuid: "cedd592820f84c90b03484926a0c18c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" is_pre_election: true
I20260929 02:09:04.616618 25172 raft_consensus.cc:2454] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedd592820f84c90b03484926a0c18c6 in term 0.
I20260929 02:09:04.616878 25307 raft_consensus.cc:2454] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedd592820f84c90b03484926a0c18c6 in term 0.
I20260929 02:09:04.617326 24962 leader_election.cc:302] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9778426f6443119cf57a0720f57ba7, cedd592820f84c90b03484926a0c18c6; no voters:
I20260929 02:09:04.617695 25386 raft_consensus.cc:2781] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:09:04.617846 25386 raft_consensus.cc:482] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:09:04.617959 25386 raft_consensus.cc:3037] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:04.620195 25386 raft_consensus.cc:504] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } }
I20260929 02:09:04.620826 25386 leader_election.cc:288] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [CANDIDATE]: Term 1 election: Requested vote from peers 2d374d98d12b42d6a9c4f1bfcf717557 (127.11.182.195:45079), 2d9778426f6443119cf57a0720f57ba7 (127.11.182.194:40145)
I20260929 02:09:04.621268 25307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6809945a976423aac08caa52e5d762e" candidate_uuid: "cedd592820f84c90b03484926a0c18c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d374d98d12b42d6a9c4f1bfcf717557"
I20260929 02:09:04.621349 25172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6809945a976423aac08caa52e5d762e" candidate_uuid: "cedd592820f84c90b03484926a0c18c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9778426f6443119cf57a0720f57ba7"
I20260929 02:09:04.621477 25307 raft_consensus.cc:3037] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:04.621537 25172 raft_consensus.cc:3037] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:04.624643 25172 raft_consensus.cc:2454] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedd592820f84c90b03484926a0c18c6 in term 1.
I20260929 02:09:04.624648 25307 raft_consensus.cc:2454] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedd592820f84c90b03484926a0c18c6 in term 1.
I20260929 02:09:04.625051 24962 leader_election.cc:302] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d374d98d12b42d6a9c4f1bfcf717557, cedd592820f84c90b03484926a0c18c6; no voters:
I20260929 02:09:04.625327 25386 raft_consensus.cc:2781] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:09:04.625854 25386 raft_consensus.cc:686] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 1 LEADER]: Becoming Leader. State: Replica: cedd592820f84c90b03484926a0c18c6, State: Running, Role: LEADER
I20260929 02:09:04.626363 25386 consensus_queue.cc:237] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } }
I20260929 02:09:04.631569 24876 catalog_manager.cc:5495] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cedd592820f84c90b03484926a0c18c6 (127.11.182.193). New cstate: current_term: 1 leader_uuid: "cedd592820f84c90b03484926a0c18c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cedd592820f84c90b03484926a0c18c6" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 } health_report { overall_health: UNKNOWN } } }
W20260929 02:09:04.636422 25353 tablet.cc:2369] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:09:04.653801 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.647809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.11.182.193:45987)
0929 02:09:04.647994 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.648000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.648020 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:09:04.648151 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.648156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.648273 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.648422 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:09:04.648751 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.649892 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.651328 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.652019 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.652164 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.652613 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:09:04.653267 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:09:04.653405 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.653451 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.653528 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:09:04.661803 25307 raft_consensus.cc:1261] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [term 1 FOLLOWER]: Refusing update from remote peer cedd592820f84c90b03484926a0c18c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:04.661928 25172 raft_consensus.cc:1261] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [term 1 FOLLOWER]: Refusing update from remote peer cedd592820f84c90b03484926a0c18c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:04.662762 25386 consensus_queue.cc:1035] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d374d98d12b42d6a9c4f1bfcf717557" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:04.663087 25390 consensus_queue.cc:1035] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d9778426f6443119cf57a0720f57ba7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:04.674722 25393 mvcc.cc:205] Tried to move back new op lower bound from 7334493162125156352 to 7334493161994379264. Current Snapshot: MvccSnapshot[applied={T|T < 7334493162125156352}]
I20260929 02:09:04.701574 24876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:09:04.701864 24876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:09:04.706251 24876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 788c52b5896648f7be8f55d8dc9f6b11: Sending DeleteTablet for 3 replicas of tablet f6809945a976423aac08caa52e5d762e
I20260929 02:09:04.707862 25287 tablet_service.cc:1514] Processing DeleteTablet for tablet f6809945a976423aac08caa52e5d762e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:04 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:60204
I20260929 02:09:04.707970 25152 tablet_service.cc:1514] Processing DeleteTablet for tablet f6809945a976423aac08caa52e5d762e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:04 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:52766
I20260929 02:09:04.709177 25008 tablet_service.cc:1514] Processing DeleteTablet for tablet f6809945a976423aac08caa52e5d762e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:04 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:47220
I20260929 02:09:04.715714 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 24913
I20260929 02:09:04.727965 25403 tablet_replica.cc:330] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: stopping tablet replica
I20260929 02:09:04.728547 25403 raft_consensus.cc:2229] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:09:04.729193 25403 raft_consensus.cc:2258] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:04.733206 25405 tablet_replica.cc:330] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: stopping tablet replica
I20260929 02:09:04.734431 25405 raft_consensus.cc:2229] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:09:04.734069 25404 tablet_replica.cc:330] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: stopping tablet replica
I20260929 02:09:04.735850 25404 raft_consensus.cc:2229] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:09:04.736624 25405 raft_consensus.cc:2258] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:04.736670 25403 ts_tablet_manager.cc:1905] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:04.737140 25404 raft_consensus.cc:2258] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:04.739287 25404 ts_tablet_manager.cc:1905] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:04.741039 25405 ts_tablet_manager.cc:1905] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:04.743126 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.721905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.11.182.193:45987)
0929 02:09:04.722278 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.722286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.722308 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:09:04.722369 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.722374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.722448 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.722546 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:04.724699 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.726161 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.729080 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.729814 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.729957 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.733853 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:04.733867 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:04.733885 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:04.733927 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:04.738483 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.739441 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.739449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.739454 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.739624 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.740056 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.740061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.740063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.741001 (+ 938us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:04.741300 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.742483 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.742725 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260929 02:09:04.743788 25068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:04.747403 25403 ts_tablet_manager.cc:1918] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:04.747562 25403 log.cc:1196] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/f6809945a976423aac08caa52e5d762e
I20260929 02:09:04.748068 25403 ts_tablet_manager.cc:1939] T f6809945a976423aac08caa52e5d762e P 2d374d98d12b42d6a9c4f1bfcf717557: Deleting consensus metadata
I20260929 02:09:04.749811 24862 catalog_manager.cc:4841] TS 2d374d98d12b42d6a9c4f1bfcf717557 (127.11.182.195:45079): tablet f6809945a976423aac08caa52e5d762e (table test-table [id=d86873a612f24fbb9400a7faf8d0ec0d]) successfully deleted
I20260929 02:09:04.750625 25404 ts_tablet_manager.cc:1918] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:04.750761 25404 log.cc:1196] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/f6809945a976423aac08caa52e5d762e
I20260929 02:09:04.751245 25404 ts_tablet_manager.cc:1939] T f6809945a976423aac08caa52e5d762e P 2d9778426f6443119cf57a0720f57ba7: Deleting consensus metadata
I20260929 02:09:04.752887 24862 catalog_manager.cc:4841] TS 2d9778426f6443119cf57a0720f57ba7 (127.11.182.194:40145): tablet f6809945a976423aac08caa52e5d762e (table test-table [id=d86873a612f24fbb9400a7faf8d0ec0d]) successfully deleted
I20260929 02:09:04.846306 25405 ts_tablet_manager.cc:1918] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:04.846514 25405 log.cc:1196] T f6809945a976423aac08caa52e5d762e P cedd592820f84c90b03484926a0c18c6: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/f6809945a976423aac08caa52e5d762e
I20260929 02:09:04.846674 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 24913
I20260929 02:09:04.864737 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 25077
W20260929 02:09:04.864727 24861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.182.193:45987 (error 108)
I20260929 02:09:04.866591 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.866110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45987 (local address 127.0.0.1:47252)
0929 02:09:04.866316 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.866436 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:09:04.866798 24861 catalog_manager.cc:4584] TS cedd592820f84c90b03484926a0c18c6 (127.11.182.193:45987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6809945a976423aac08caa52e5d762e: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45987: connect: Connection refused (error 111)
I20260929 02:09:04.878393 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.865579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.11.182.194:40145)
0929 02:09:04.865764 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.865770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.865868 (+ 98us) server_negotiation.cc:407] Connection header received
0929 02:09:04.866079 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.866083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.866173 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.866278 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:04.867821 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.868604 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.869846 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.870258 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.870353 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.872937 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:04.872950 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:04.872965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:04.873002 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:04.875916 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.876570 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.876576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.876578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.876719 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:04.877076 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:04.877079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:04.877081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:04.877411 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:04.877644 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:04.877947 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 02:09:04.878108 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:09:04.879180 25212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:04.920487 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.919919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45987 (local address 127.0.0.1:47256)
0929 02:09:04.920156 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.920280 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:09:04.921612 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.921234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45987 (local address 127.0.0.1:47266)
0929 02:09:04.921380 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.921467 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 02:09:04.970952 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 25077
I20260929 02:09:04.982046 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.981459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45987 (local address 127.0.0.1:47270)
0929 02:09:04.981712 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.981833 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:09:04.983234 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.982715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45987 (local address 127.0.0.1:47272)
0929 02:09:04.982923 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:04.983044 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 02:09:04.989142 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 25222
I20260929 02:09:05.002161 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:04.989882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.11.182.195:45079)
0929 02:09:04.990041 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:04.990046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:04.990064 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:09:04.990203 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:04.990207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:04.990266 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:04.990356 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:04.991780 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.992509 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.993825 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:04.994220 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:04.994312 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:04.997026 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:04.997040 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:04.997056 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:04.997085 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:04.999873 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:05.000482 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:05.000489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:05.000491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:05.000601 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:05.000905 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:05.000908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:05.000910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:05.001248 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:05.001486 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:05.001738 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 02:09:05.001897 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:09:05.002839 25347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:05.063988 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:05.063446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45987 (local address 127.0.0.1:47274)
0929 02:09:05.063699 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:05.063811 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 02:09:05.065838 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:05.065164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45987 (local address 127.0.0.1:47278)
0929 02:09:05.065554 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:05.065655 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279}
I20260929 02:09:05.092954 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 25222
I20260929 02:09:05.110252 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 24843
I20260929 02:09:05.111603 24906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:05.225349 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 24843
I20260929 02:09:05.247911 11995 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:09:05.583791 11995 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:09:05.584033 11995 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:09:05.930117 11995 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:09:05.930 UTC [23894] LOG: received smart shutdown request
2026-09-29 02:09:05.933 UTC [23894] LOG: background worker "logical replication launcher" (PID 23903) exited with exit code 1
2026-09-29 02:09:05.934 UTC [23898] LOG: shutting down
2026-09-29 02:09:06.114 UTC [23894] LOG: database system is shut down
2026-09-29T02:09:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52278 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:09:06 dist-test-slave-q06p krb5kdc[25419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:09:06 dist-test-slave-q06p krb5kdc[25419](info): set up 2 sockets
Sep 29 02:09:06 dist-test-slave-q06p krb5kdc[25419](info): commencing operation
krb5kdc: starting...
W20260929 02:09:07.347867 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:09:07 dist-test-slave-q06p krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:09:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:09:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:09:07.582273 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:43975
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:43975,127.11.182.253:35559,127.11.182.252:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:07.921988 25432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:07.922394 25432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:07.922571 25432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:07.930120 25432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:09:07.930289 25432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:07.930415 25432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:07.930531 25432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:09:07.930625 25432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:09:07.940927 25432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:43975,127.11.182.253:35559,127.11.182.252:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:43975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:07.942368 25432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:07.943725 25432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:07.955996 25437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:07.956131 25438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:07.956154 25440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:07.957270 25432 server_base.cc:1034] running on GCE node
I20260929 02:09:07.958142 25432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:07.959774 25432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:07.961007 25432 hybrid_clock.cc:648] HybridClock initialized: now 1790647747960971 us; error 38 us; skew 500 ppm
I20260929 02:09:07.961366 25432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:07 dist-test-slave-q06p krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647747, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:07.968087 25432 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:09:07.969836 25432 webserver.cc:460] Webserver started at http://127.11.182.254:46735/ using document root <none> and password file <none>
I20260929 02:09:07.970623 25432 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:07.970747 25432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:07.971180 25432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:07.974300 25432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "8979be1b4302453d9335391bff6d7980"
format_stamp: "Formatted at 2026-09-29 02:09:07 on dist-test-slave-q06p"
server_key: "27f9746611d581a7062ceb1712e20856"
server_key_iv: "b687c23b1f9b4a7b360e1ed1d93fc731"
server_key_version: "encryptionkey@0"
I20260929 02:09:07.975193 25432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "8979be1b4302453d9335391bff6d7980"
format_stamp: "Formatted at 2026-09-29 02:09:07 on dist-test-slave-q06p"
server_key: "27f9746611d581a7062ceb1712e20856"
server_key_iv: "b687c23b1f9b4a7b360e1ed1d93fc731"
server_key_version: "encryptionkey@0"
I20260929 02:09:07.980512 25432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:09:07.984457 25447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:07.985908 25432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:09:07.986097 25432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "8979be1b4302453d9335391bff6d7980"
format_stamp: "Formatted at 2026-09-29 02:09:07 on dist-test-slave-q06p"
server_key: "27f9746611d581a7062ceb1712e20856"
server_key_iv: "b687c23b1f9b4a7b360e1ed1d93fc731"
server_key_version: "encryptionkey@0"
I20260929 02:09:07.986296 25432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:08.005671 25432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:08.010187 25432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:08.010500 25432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:08.030761 25432 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:43975
I20260929 02:09:08.030812 25498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:43975 every 8 connection(s)
I20260929 02:09:08.032300 25432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:09:08.035432 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 25432
I20260929 02:09:08.035696 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:09:08.036567 11995 external_mini_cluster.cc:1428] Setting key 0dd35e4c3bffab8d2c06c13d38c8227c
I20260929 02:09:08.036864 25499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:08.047343 25499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.182.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.253@KRBTEST.COM" created.
Sep 29 02:09:08 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647747, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:08.074057 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.051148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43975 (local address 127.0.0.1:39566)
0929 02:09:08.052376 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.052450 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:08.054453 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:08.055509 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:08.055514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:08.056081 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:08.056418 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:08.056435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.059258 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.059286 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:08.060741 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:08.060754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.062348 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.062354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:08.062484 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:08.063282 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:08.063308 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:08.066113 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:08.069585 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:08.069597 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:08.069610 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:08.070007 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:08.070477 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:08.070482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:08.070486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:08.070635 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:08.071518 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:08.071527 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:08.071770 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 02:09:08.072038 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 02:09:08.072642 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":166,"thread_start_us":549,"threads_started":1}
I20260929 02:09:08.074091 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.051148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.11.182.254:43975)
0929 02:09:08.053207 (+ 2059us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:08.053219 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:08.053280 (+ 61us) server_negotiation.cc:407] Connection header received
0929 02:09:08.054892 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:08.054903 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:08.055245 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:08.055478 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:08.056644 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.059015 (+ 2371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.060894 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.062080 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.062257 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:08.066353 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:08.066361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:08.066374 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:08.066403 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:08.069413 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:08.070165 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:08.070177 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:08.070179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:08.070354 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:08.070741 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:08.070748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:08.070752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:08.071352 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:08.071588 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:08.072140 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 02:09:08.072526 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1805,"spinlock_wait_cycles":38656,"thread_start_us":1580,"threads_started":1}
I20260929 02:09:08.077178 25499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:09:08.078768 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.078091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52482)
0929 02:09:08.078428 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.078571 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
W20260929 02:09:08.079106 25450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
I20260929 02:09:08.081328 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.080982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52496)
0929 02:09:08.081144 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.081214 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:09:08.081653 25499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:35559: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab.
Entry for principal kudu/127.11.182.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab.
I20260929 02:09:08.095444 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.253:35559
--webserver_interface=127.11.182.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab
--principal=kudu/127.11.182.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:43975,127.11.182.253:35559,127.11.182.252:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:09:08.131033 25499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } attempt: 1
I20260929 02:09:08.132357 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.131857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52498)
0929 02:09:08.132075 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.132198 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 02:09:08.133414 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.133070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52508)
0929 02:09:08.133209 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.133277 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:09:08.133657 25499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:35559: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
I20260929 02:09:08.201937 25499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } attempt: 2
I20260929 02:09:08.203241 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.202694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52518)
0929 02:09:08.202985 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.203086 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 02:09:08.204280 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.203947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52532)
0929 02:09:08.204075 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.204167 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:09:08.204516 25499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:35559: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
I20260929 02:09:08.295795 25499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } attempt: 3
I20260929 02:09:08.296980 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.296551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52536)
0929 02:09:08.296748 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.296854 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 02:09:08.298075 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.297718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52552)
0929 02:09:08.297878 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.297946 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 02:09:08.298374 25499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:35559: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
W20260929 02:09:08.433640 25506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:08.433965 25506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:08.434132 25506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 02:09:08.441653 25499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } attempt: 4
W20260929 02:09:08.441869 25506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:09:08.442054 25506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:08.442139 25506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:08.442229 25506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:09:08.442304 25506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:09:08.443028 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.442417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52566)
0929 02:09:08.442732 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.442827 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260929 02:09:08.444051 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.443729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52582)
0929 02:09:08.443853 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.443934 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:09:08.444299 25499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:35559: Network error: Client connection negotiation failed: client connection to 127.11.182.253:35559: connect: Connection refused (error 111)
I20260929 02:09:08.453379 25506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:43975,127.11.182.253:35559,127.11.182.252:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab
--principal=kudu/127.11.182.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.253:35559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.182.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:08.454833 25506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:08.456239 25506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:08.468698 25512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:08.468792 25511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:08.468768 25514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:08.469563 25506 server_base.cc:1034] running on GCE node
I20260929 02:09:08.470350 25506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:08.471935 25506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:08.473167 25506 hybrid_clock.cc:648] HybridClock initialized: now 1790647748473114 us; error 48 us; skew 500 ppm
I20260929 02:09:08.473521 25506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:08 dist-test-slave-q06p krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647748, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:08.476799 25506 init.cc:377] Logged in from keytab as kudu/127.11.182.253@KRBTEST.COM (short username kudu)
I20260929 02:09:08.478380 25506 webserver.cc:460] Webserver started at http://127.11.182.253:45509/ using document root <none> and password file <none>
I20260929 02:09:08.479171 25506 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:08.479298 25506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:08.479559 25506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:08.482563 25506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/data/instance:
uuid: "fbe3f783dad64dbba10f104df00e7de5"
format_stamp: "Formatted at 2026-09-29 02:09:08 on dist-test-slave-q06p"
server_key: "c508ca2c6dcc2bb70770ccd137ff3ac2"
server_key_iv: "f3bd50f6ce1093d0432b9e505ac8ad92"
server_key_version: "encryptionkey@0"
I20260929 02:09:08.483372 25506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/wal/instance:
uuid: "fbe3f783dad64dbba10f104df00e7de5"
format_stamp: "Formatted at 2026-09-29 02:09:08 on dist-test-slave-q06p"
server_key: "c508ca2c6dcc2bb70770ccd137ff3ac2"
server_key_iv: "f3bd50f6ce1093d0432b9e505ac8ad92"
server_key_version: "encryptionkey@0"
I20260929 02:09:08.488725 25506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:09:08.492280 25521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:08.493619 25506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:09:08.493803 25506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/wal
uuid: "fbe3f783dad64dbba10f104df00e7de5"
format_stamp: "Formatted at 2026-09-29 02:09:08 on dist-test-slave-q06p"
server_key: "c508ca2c6dcc2bb70770ccd137ff3ac2"
server_key_iv: "f3bd50f6ce1093d0432b9e505ac8ad92"
server_key_version: "encryptionkey@0"
I20260929 02:09:08.493996 25506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:08.510743 25506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:08.515268 25506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:08.515547 25506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:08.534605 25506 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.253:35559
I20260929 02:09:08.534634 25572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.253:35559 every 8 connection(s)
I20260929 02:09:08.536065 25506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
I20260929 02:09:08.538192 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 25506
I20260929 02:09:08.538378 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-1/wal/instance
I20260929 02:09:08.538934 11995 external_mini_cluster.cc:1428] Setting key ef22e00647e6019d2d5ae6fb1dd510e8
I20260929 02:09:08.540006 25573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:08.546638 25573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.182.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.252@KRBTEST.COM" created.
Sep 29 02:09:08 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647748, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:08.570879 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.549393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.11.182.254:43975)
0929 02:09:08.549617 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:08.549623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:08.550984 (+ 1361us) server_negotiation.cc:407] Connection header received
0929 02:09:08.554038 (+ 3054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:08.554043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:08.554123 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:08.554215 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:08.556483 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.557173 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.559783 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.560338 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.560453 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:08.564636 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:08.564642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:08.564645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:08.564680 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:08.567591 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:08.568414 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:08.568419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:08.568421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:08.568546 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:08.569003 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:08.569010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:08.569018 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:08.569309 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:08.569485 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:08.570259 (+ 774us) server_negotiation.cc:299] Negotiation successful
0929 02:09:08.570505 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:09:08.571969 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.549875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43975 (local address 127.0.0.1:39570)
0929 02:09:08.550793 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.550866 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:08.553712 (+ 2846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:08.554490 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:08.554506 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:08.555378 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:08.556271 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:08.556300 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.557349 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.557358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:08.559599 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:08.559624 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.560583 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.560588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:08.560775 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:08.561491 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:08.561515 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:08.564378 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:08.567756 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:08.567765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:08.567785 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:08.568259 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:08.568685 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:08.568690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:08.568694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:08.568877 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:08.569453 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:08.569463 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:08.569872 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 02:09:08.570154 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 02:09:08.570745 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":339,"threads_started":1}
I20260929 02:09:08.573872 25573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:09:08 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647748, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:09:08.589836 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.574727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52594)
0929 02:09:08.574967 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.574990 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:08.575152 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:08.576028 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:08.576032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:08.576439 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:08.576586 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:08.576603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.578623 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.578627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:08.579657 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:08.579669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.580993 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.580997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:08.581164 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:08.581818 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:08.581835 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:08.583866 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:08.587365 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:08.587369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:08.587372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:08.587754 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:08.588182 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:08.588185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:08.588188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:08.588344 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:08.588938 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:08.588942 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:08.589112 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 02:09:08.589239 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 02:09:08.589445 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 02:09:08.590240 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.574795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.11.182.253:35559)
0929 02:09:08.575492 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:08.575503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:08.575537 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:09:08.575600 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:08.575611 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:08.575847 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:08.576012 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:08.576760 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.578453 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.579804 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.580815 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.580959 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:08.584118 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:08.584133 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:08.584152 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:08.584193 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:08.587184 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:08.587924 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:08.587937 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:08.587945 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:08.588081 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:08.588460 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:08.588464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:08.588466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:08.588823 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:08.588976 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:08.589425 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 02:09:08.589736 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":313,"threads_started":1}
I20260929 02:09:08.592772 25573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab.
Entry for principal kudu/127.11.182.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab.
I20260929 02:09:08.594007 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.593577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33868)
0929 02:09:08.593742 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.593861 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 02:09:08.594271 25522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
I20260929 02:09:08.594362 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.252:33401
--webserver_interface=127.11.182.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab
--principal=kudu/127.11.182.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:43975,127.11.182.253:35559,127.11.182.252:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:09:08.595961 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.595595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33874)
0929 02:09:08.595786 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.595861 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 02:09:08.596215 25573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:33401: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
I20260929 02:09:08.645576 25573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } attempt: 1
I20260929 02:09:08.646929 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.646350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33876)
0929 02:09:08.646581 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.646693 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 02:09:08.647980 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.647589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33880)
0929 02:09:08.647793 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.647871 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:09:08.648192 25573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:33401: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
I20260929 02:09:08.716464 25573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } attempt: 2
I20260929 02:09:08.717620 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.717157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33896)
0929 02:09:08.717361 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.717456 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 02:09:08.718552 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.718235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33910)
0929 02:09:08.718378 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.718447 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:09:08.718762 25573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:33401: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
I20260929 02:09:08.743610 25499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } attempt: 5
Sep 29 02:09:08 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647747, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:09:08.758685 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.744642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.11.182.253:35559)
0929 02:09:08.744816 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:08.744821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:08.744840 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:09:08.744894 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:08.744898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:08.744977 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:08.745066 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:08.746174 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.746933 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.748958 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.749529 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.749630 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:08.753343 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:08.753348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:08.753352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:08.753384 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:08.756064 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:08.756749 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:08.756757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:08.756760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:08.756883 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:08.757254 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:08.757260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:08.757264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:08.757581 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:08.757745 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:08.758112 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 02:09:08.758323 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:09:08.758692 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.744390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52608)
0929 02:09:08.744622 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.744644 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:08.744786 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:08.745132 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:08.745138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:08.745751 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:08.745992 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:08.746005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.747085 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.747096 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:08.748784 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:08.748806 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:08.749691 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:08.749699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:08.749885 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:08.750541 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:08.750564 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:08.753139 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:08.756226 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:08.756231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:08.756236 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:08.756625 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:08.756997 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:08.757002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:08.757006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:08.757148 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:08.757706 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:08.757711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:08.757872 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 02:09:08.758014 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 02:09:08.758322 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:09:08.759806 25499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:09:08.760986 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.760516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33916)
0929 02:09:08.760721 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.760818 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:09:08.761987 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.761622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33924)
0929 02:09:08.761771 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.761866 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:09:08.762199 25499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:33401: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
I20260929 02:09:08.809989 25573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } attempt: 3
I20260929 02:09:08.811272 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.810747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33936)
0929 02:09:08.811015 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.811116 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 02:09:08.812593 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.811968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33948)
0929 02:09:08.812291 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.812421 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
W20260929 02:09:08.812865 25573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:33401: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
I20260929 02:09:08.813431 25499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } attempt: 1
I20260929 02:09:08.814520 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.814028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33956)
0929 02:09:08.814233 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.814345 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:09:08.815668 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.815206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33972)
0929 02:09:08.815390 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.815498 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:09:08.815907 25499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:33401: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
I20260929 02:09:08.884166 25499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } attempt: 2
I20260929 02:09:08.885535 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.884943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33988)
0929 02:09:08.885220 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.885334 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:09:08.886720 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.886278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:33994)
0929 02:09:08.886481 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.886576 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 02:09:08.887022 25499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:33401: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
W20260929 02:09:08.933720 25580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:08.934063 25580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:08.934221 25580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:08.941592 25580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:09:08.941751 25580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:08.941864 25580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:08.941970 25580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:09:08.942054 25580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:09:08.952291 25580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:43975,127.11.182.253:35559,127.11.182.252:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab
--principal=kudu/127.11.182.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.252:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.182.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:08.953647 25580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:08.955010 25580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 02:09:08.956099 25573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } attempt: 4
I20260929 02:09:08.957347 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.956800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:34000)
0929 02:09:08.957047 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.957150 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 02:09:08.958467 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.958072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:34006)
0929 02:09:08.958249 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.958339 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:09:08.958698 25573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:33401: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
W20260929 02:09:08.967427 25585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:08.967499 25588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:08.967459 25586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:08.968520 25580 server_base.cc:1034] running on GCE node
I20260929 02:09:08.969269 25580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:08.970621 25580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:08.971863 25580 hybrid_clock.cc:648] HybridClock initialized: now 1790647748971789 us; error 81 us; skew 500 ppm
I20260929 02:09:08.972225 25580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:08 dist-test-slave-q06p krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647748, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:08.975286 25580 init.cc:377] Logged in from keytab as kudu/127.11.182.252@KRBTEST.COM (short username kudu)
I20260929 02:09:08.976768 25580 webserver.cc:460] Webserver started at http://127.11.182.252:43469/ using document root <none> and password file <none>
I20260929 02:09:08.977492 25580 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:08.977640 25580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:08.977941 25580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:08.981125 25580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/data/instance:
uuid: "f278777a43f6489c9938493ba896ab65"
format_stamp: "Formatted at 2026-09-29 02:09:08 on dist-test-slave-q06p"
server_key: "47ff1b8eb598244e6cf672f064bcb920"
server_key_iv: "c53742bc2dd12d1519290ac61171140c"
server_key_version: "encryptionkey@0"
I20260929 02:09:08.981940 25580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/wal/instance:
uuid: "f278777a43f6489c9938493ba896ab65"
format_stamp: "Formatted at 2026-09-29 02:09:08 on dist-test-slave-q06p"
server_key: "47ff1b8eb598244e6cf672f064bcb920"
server_key_iv: "c53742bc2dd12d1519290ac61171140c"
server_key_version: "encryptionkey@0"
I20260929 02:09:08.987404 25580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:09:08.991067 25595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:08.992429 25580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:09:08.992610 25580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/wal
uuid: "f278777a43f6489c9938493ba896ab65"
format_stamp: "Formatted at 2026-09-29 02:09:08 on dist-test-slave-q06p"
server_key: "47ff1b8eb598244e6cf672f064bcb920"
server_key_iv: "c53742bc2dd12d1519290ac61171140c"
server_key_version: "encryptionkey@0"
I20260929 02:09:08.992806 25580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:08.993237 25499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } attempt: 3
I20260929 02:09:08.994345 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.993903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:34008)
0929 02:09:08.994111 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.994196 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 02:09:08.995385 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:08.995009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:34020)
0929 02:09:08.995221 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:08.995283 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 02:09:08.995620 25499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:33401: Network error: Client connection negotiation failed: client connection to 127.11.182.252:33401: connect: Connection refused (error 111)
I20260929 02:09:09.016989 25580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:09.021489 25580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:09.021796 25580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:09.040728 25646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.252:33401 every 8 connection(s)
I20260929 02:09:09.040750 25580 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.252:33401
I20260929 02:09:09.042503 25580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
I20260929 02:09:09.046442 25647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:09.048303 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 25580
I20260929 02:09:09.048534 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647540175441-11995-0/minicluster-data/master-2/wal/instance
I20260929 02:09:09.049096 11995 external_mini_cluster.cc:1428] Setting key 6dd531a49fb20e6446dc58da4e96930a
I20260929 02:09:09.056350 25647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:09:09 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
Sep 29 02:09:09 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647748, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:09.072790 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.051162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.11.182.252:33401)
0929 02:09:09.052008 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.052019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.052064 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:09:09.053676 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.053698 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.054103 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.054423 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.056225 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.057659 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.059489 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.060311 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.060486 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.065269 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.065277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.065289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.065330 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.068560 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.069279 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.069286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.069292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.069437 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.069818 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.069822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.069824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.070355 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.070657 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.071112 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.071527 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":412,"threads_started":1}
I20260929 02:09:09.074007 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.059048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43975 (local address 127.0.0.1:39574)
0929 02:09:09.059862 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:09.059910 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:09.060061 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:09.060479 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:09.060486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:09.061342 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:09.061840 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.061863 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.062909 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.062917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.064299 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.064316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.065129 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.065150 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.065310 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.065985 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:09.066001 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:09.068300 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:09.071524 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.071538 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.071543 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.071972 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.072392 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.072395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.072397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.072524 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.073164 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:09.073169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:09.073330 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 02:09:09.073485 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 02:09:09.073726 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":408,"threads_started":1}
I20260929 02:09:09.074230 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.059048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.11.182.254:43975)
0929 02:09:09.059267 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.059275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.060013 (+ 738us) server_negotiation.cc:407] Connection header received
0929 02:09:09.060187 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.060193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.060278 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.060386 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.062030 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.062712 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.064446 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.064873 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.064981 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.068519 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.068526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.068530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.068569 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.071354 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.072111 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.072116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.072119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.072278 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.072651 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.072656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.072659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.073010 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.073172 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.073644 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.073868 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:09:09.075448 25647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:09:09 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647748, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
Sep 29 02:09:09 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:09.092281 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.076220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:35559 (local address 127.0.0.1:52616)
0929 02:09:09.076447 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:09.076465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:09.076557 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:09.077134 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:09.077139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:09.077485 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:09.077624 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.077632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.079012 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.079016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.080419 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.080439 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.081465 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.081470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.081729 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.082363 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:09.082388 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:09.085500 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:09.089621 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.089627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.089631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.090068 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.090529 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.090532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.090535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.090774 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.091355 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:09.091362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:09.091533 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 02:09:09.091705 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 02:09:09.092040 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260929 02:09:09.092612 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.076679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.11.182.253:35559)
0929 02:09:09.076853 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.076859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.076875 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:09:09.076930 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.076935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.077002 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.077106 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.077771 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.078868 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.080578 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.081147 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.081264 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.085742 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.085752 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.085758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.085800 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.089360 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.090196 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.090204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.090210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.090394 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.090925 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.090930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.090933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.091236 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.091383 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.091811 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.092025 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:09:09.093849 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.076376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.11.182.254:43975)
0929 02:09:09.076555 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.076567 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.076587 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:09:09.076639 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.076644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.076710 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.076802 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.078026 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.079142 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.080849 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.081483 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.081619 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.086796 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.086804 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.086811 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.086881 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.090483 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.091411 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.091416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.091420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.091593 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.092006 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.092011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.092014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.092373 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.092664 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.093225 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.093485 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:09:09.093814 25647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:09:09 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647748, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
Sep 29 02:09:09 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:09:09.110399 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.094733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.11.182.252:33401)
0929 02:09:09.094927 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.094934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.094952 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:09:09.094993 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.094998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.095070 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.095160 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.096010 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.097044 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.098576 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.099324 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.099455 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.105111 (+ 5656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.105116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.105119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.105147 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.107924 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.108537 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.108545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.108548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.108666 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.109029 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.109040 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.109043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.109311 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.109460 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.109794 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.110011 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:09:09.110587 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.094529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:34030)
0929 02:09:09.094697 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:09.094731 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:09.094857 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:09.095183 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:09.095186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:09.095672 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:09.095859 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.095870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.097176 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.097184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.098437 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.098449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.099478 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.099485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.099742 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.100393 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:09.100420 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:09.102793 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:09.108068 (+ 5275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.108074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.108078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.108425 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.108769 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.108776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.108782 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.108919 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.109423 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:09.109428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:09.109567 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 02:09:09.109716 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 02:09:09.110166 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":139}
I20260929 02:09:09.112286 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.095754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.11.182.253:35559)
0929 02:09:09.095907 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.095911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.095963 (+ 52us) server_negotiation.cc:407] Connection header received
0929 02:09:09.096145 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.096152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.096237 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.096341 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.098013 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.099279 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.101058 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.101497 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.101595 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.106529 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.106535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.106539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.106572 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.109387 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.110130 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.110138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.110142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.110310 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.110753 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.110760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.110765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.111117 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.111330 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.111715 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.111965 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:09:09.116513 25647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65: Bootstrap starting.
I20260929 02:09:09.117348 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:09:09.120744 25647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:09.122254 25647 log.cc:824] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:09.126228 25647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65: No bootstrap required, opened a new log
I20260929 02:09:09.138144 25647 raft_consensus.cc:348] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } }
I20260929 02:09:09.138661 25647 raft_consensus.cc:374] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:09.138813 25647 raft_consensus.cc:729] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f278777a43f6489c9938493ba896ab65, State: Initialized, Role: FOLLOWER
I20260929 02:09:09.139765 25647 consensus_queue.cc:260] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } }
I20260929 02:09:09.144636 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.124012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.11.182.254:43975)
0929 02:09:09.124238 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.124246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.124335 (+ 89us) server_negotiation.cc:407] Connection header received
0929 02:09:09.124657 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.124662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.124748 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.124820 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.127377 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.128489 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.130690 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.131388 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.131499 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.135786 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.135794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.135799 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.135854 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.140147 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.141308 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.141317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.141324 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.141550 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.142175 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.142183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.142189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.142676 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.142986 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.143763 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.144098 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:09:09.144809 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.123470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.11.182.253:35559)
0929 02:09:09.123662 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.123668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.124863 (+ 1195us) server_negotiation.cc:407] Connection header received
0929 02:09:09.125598 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.125606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.125707 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.125793 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.128719 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.130162 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.133143 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.133849 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.133955 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.137542 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.137550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.137555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.137602 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.141311 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.142265 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.142273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.142279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.142461 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.142991 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.142998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.143002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.143394 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.143634 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.144140 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.144386 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:09:09.147480 25660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.147958 25660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:09:09.148864 25647 sys_catalog.cc:564] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:09:09.150478 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.125194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.11.182.252:33401)
0929 02:09:09.125445 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.125454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.132641 (+ 7187us) server_negotiation.cc:407] Connection header received
0929 02:09:09.132907 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.132914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.132993 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.133078 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.135337 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.136617 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.138997 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.139858 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.139967 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.143683 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.143690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.143694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.143729 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.147590 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.148337 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.148342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.148346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.148482 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.148895 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.148900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.148903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.149264 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.149546 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.149846 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.150060 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 02:09:09.163277 25671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:09:09.163426 25671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:09:09.160215 25580 master_runner.cc:186] Error getting master registration for 127.11.182.254:43975: OK, instance_id { permanent_uuid: "8979be1b4302453d9335391bff6d7980" instance_seqno: 1790647748024815 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:09:09.172456 25580 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:09:09.172818 25499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } attempt: 4
Sep 29 02:09:09 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647747, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
I20260929 02:09:09.188222 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.173490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.11.182.252:33401)
0929 02:09:09.173726 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.173733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.173753 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:09:09.173864 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.173870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.173948 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.174070 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.174943 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.175805 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.177208 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.177731 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.177900 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.182202 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.182209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.182215 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.182260 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.185272 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.185965 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.185971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.185975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.186127 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.186545 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.186551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.186555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.186966 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.187162 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.187573 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.187839 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 02:09:09.188242 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.173425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:34052)
0929 02:09:09.173615 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:09.173637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:09.173750 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:09.174097 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:09.174102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:09.174561 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:09.174743 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.174752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.175947 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.175954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.177059 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.177072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.177909 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.177915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.178116 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.178809 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:09.178829 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:09.181997 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:09.185427 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.185433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.185438 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.185825 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.186253 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.186261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.186265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.186418 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.187113 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:09.187118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:09.187288 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 02:09:09.187445 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 02:09:09.187777 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 02:09:09.193744 25499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980: Bootstrap starting.
I20260929 02:09:09.197474 25499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:09.198899 25499 log.cc:824] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:09.202416 25499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980: No bootstrap required, opened a new log
I20260929 02:09:09.210691 25499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } }
I20260929 02:09:09.211179 25499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:09.211331 25499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8979be1b4302453d9335391bff6d7980, State: Initialized, Role: FOLLOWER
I20260929 02:09:09.211946 25499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } }
I20260929 02:09:09.213020 25674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.213472 25674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:09:09.214576 25499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:09:09.225616 25685 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:09:09.225769 25685 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:09:09.224695 25432 master_runner.cc:186] Error getting master registration for 127.11.182.253:35559: OK, instance_id { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" instance_seqno: 1790647748529112 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:09:09.236634 25432 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:09:09.257939 25573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } attempt: 5
W20260929 02:09:09.263523 25453 tablet.cc:2369] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:09:09 dist-test-slave-q06p krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647748, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
W20260929 02:09:09.274060 25601 tablet.cc:2369] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:09:09.276108 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.258691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:33401 (local address 127.0.0.1:34064)
0929 02:09:09.259064 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:09.259088 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:09.259244 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:09.259804 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:09.259809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:09.260361 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:09.260589 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.260602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.262129 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.262135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.263645 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:09.263662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.264756 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.264770 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:09.265016 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.265736 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:09.265763 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:09.268769 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:09.273129 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.273136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.273141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.273646 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.274103 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:09.274109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:09.274113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:09.274295 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:09.275080 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:09.275085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:09.275251 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 02:09:09.275390 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 02:09:09.275698 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260929 02:09:09.277048 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:09.258779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34064 (local address 127.11.182.252:33401)
0929 02:09:09.258996 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:09.259003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:09.259181 (+ 178us) server_negotiation.cc:407] Connection header received
0929 02:09:09.259387 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:09.259393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:09.259479 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:09.259597 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:09.260766 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.261933 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.263830 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:09.264572 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:09.264730 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:09.269004 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:09.269010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:09.269016 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:09.269058 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:09.272946 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.273792 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.273798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.273802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.273940 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:09.274424 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:09.274429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:09.274433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:09.274807 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:09.275006 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:09.276398 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0929 02:09:09.276652 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 02:09:09.281778 25674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:09:09.282090 25674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } }
I20260929 02:09:09.283907 25674 leader_election.cc:288] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe3f783dad64dbba10f104df00e7de5 (127.11.182.253:35559), f278777a43f6489c9938493ba896ab65 (127.11.182.252:33401)
I20260929 02:09:09.285538 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8979be1b4302453d9335391bff6d7980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f278777a43f6489c9938493ba896ab65" is_pre_election: true
I20260929 02:09:09.286265 25622 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8979be1b4302453d9335391bff6d7980 in term 0.
I20260929 02:09:09.287035 25448 leader_election.cc:302] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8979be1b4302453d9335391bff6d7980, f278777a43f6489c9938493ba896ab65; no voters:
I20260929 02:09:09.287606 25674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:09:09.287774 25674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:09:09.287894 25674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:09.290310 25573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5: Bootstrap starting.
I20260929 02:09:09.291714 25674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } }
I20260929 02:09:09.294008 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8979be1b4302453d9335391bff6d7980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f278777a43f6489c9938493ba896ab65"
I20260929 02:09:09.294263 25622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:09.297627 25674 leader_election.cc:288] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [CANDIDATE]: Term 1 election: Requested vote from peers fbe3f783dad64dbba10f104df00e7de5 (127.11.182.253:35559), f278777a43f6489c9938493ba896ab65 (127.11.182.252:33401)
I20260929 02:09:09.299252 25622 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8979be1b4302453d9335391bff6d7980 in term 1.
I20260929 02:09:09.299925 25573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:09.301390 25573 log.cc:824] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:09.301771 25448 leader_election.cc:302] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8979be1b4302453d9335391bff6d7980, f278777a43f6489c9938493ba896ab65; no voters:
I20260929 02:09:09.302305 25674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:09:09.311499 25573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5: No bootstrap required, opened a new log
I20260929 02:09:09.311414 25674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [term 1 LEADER]: Becoming Leader. State: Replica: 8979be1b4302453d9335391bff6d7980, State: Running, Role: LEADER
I20260929 02:09:09.312181 25674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } }
I20260929 02:09:09.326022 25573 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } }
I20260929 02:09:09.326496 25573 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:09.326644 25573 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe3f783dad64dbba10f104df00e7de5, State: Initialized, Role: FOLLOWER
I20260929 02:09:09.326608 25688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8979be1b4302453d9335391bff6d7980. Latest consensus state: current_term: 1 leader_uuid: "8979be1b4302453d9335391bff6d7980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.327252 25688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:09.327497 25573 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } }
I20260929 02:09:09.331700 25691 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.332190 25691 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:09:09.334590 25547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8979be1b4302453d9335391bff6d7980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe3f783dad64dbba10f104df00e7de5"
I20260929 02:09:09.334524 25548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8979be1b4302453d9335391bff6d7980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe3f783dad64dbba10f104df00e7de5" is_pre_election: true
I20260929 02:09:09.335206 25547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:09.334496 25573 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:09:09.336421 25690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:09:09.338903 25690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:09:09.339140 25547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8979be1b4302453d9335391bff6d7980 in term 1.
I20260929 02:09:09.359484 25622 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [term 1 FOLLOWER]: Refusing update from remote peer 8979be1b4302453d9335391bff6d7980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:09.360702 25688 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:09.368721 25547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [term 1 FOLLOWER]: Refusing update from remote peer 8979be1b4302453d9335391bff6d7980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:09.370088 25688 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:09.385036 25660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8979be1b4302453d9335391bff6d7980. Latest consensus state: current_term: 1 leader_uuid: "8979be1b4302453d9335391bff6d7980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.385360 25660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:09:09.391006 25694 mvcc.cc:205] Tried to move back new op lower bound from 7334493181357445120 to 7334493181187747840. Current Snapshot: MvccSnapshot[applied={T|T < 7334493181357445120}]
I20260929 02:09:09.393227 25674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8979be1b4302453d9335391bff6d7980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.393671 25674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:09.399581 25690 catalog_manager.cc:1347] Generated new cluster ID: 55f190695aeb4900bb8f8a43de734a33
I20260929 02:09:09.399699 25690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:09:09.408372 25691 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8979be1b4302453d9335391bff6d7980. Latest consensus state: current_term: 1 leader_uuid: "8979be1b4302453d9335391bff6d7980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.408869 25691 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:09:09.413306 25688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8979be1b4302453d9335391bff6d7980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.413686 25688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:09.418329 25660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8979be1b4302453d9335391bff6d7980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.418694 25660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:09:09.425406 25690 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:09:09.427881 25690 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:09:09.437635 25691 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8979be1b4302453d9335391bff6d7980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979be1b4302453d9335391bff6d7980" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43975 } } peers { permanent_uuid: "fbe3f783dad64dbba10f104df00e7de5" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 35559 } } peers { permanent_uuid: "f278777a43f6489c9938493ba896ab65" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 33401 } } }
I20260929 02:09:09.438017 25691 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:09:09.458577 25690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8979be1b4302453d9335391bff6d7980: Generated new TSK 0
I20260929 02:09:09.459764 25690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:09:09.484100 25712 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:09:10.165498 25671 catalog_manager.cc:1259] Loaded cluster ID: 55f190695aeb4900bb8f8a43de734a33
I20260929 02:09:10.165597 25671 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65: loading cluster ID for follower catalog manager: success
I20260929 02:09:10.168058 25671 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65: acquiring CA information for follower catalog manager: success
I20260929 02:09:10.169759 25671 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f278777a43f6489c9938493ba896ab65: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:09:10.488686 25712 catalog_manager.cc:1259] Loaded cluster ID: 55f190695aeb4900bb8f8a43de734a33
I20260929 02:09:10.488830 25712 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5: loading cluster ID for follower catalog manager: success
I20260929 02:09:10.491802 25712 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5: acquiring CA information for follower catalog manager: success
I20260929 02:09:10.493711 25712 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbe3f783dad64dbba10f104df00e7de5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:09:10.560568 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 25432
I20260929 02:09:10.561913 25494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:10.699635 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 25432
I20260929 02:09:10.726208 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 25506
W20260929 02:09:10.726262 25525 connection.cc:535] server connection from 127.0.0.1:52608 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:09:10.727483 25568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:10.837548 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 25506
I20260929 02:09:10.857193 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 25580
I20260929 02:09:10.858261 25642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:10.965464 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 25580
2026-09-29T02:09:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4726 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:09:11 dist-test-slave-q06p krb5kdc[25723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:09:11 dist-test-slave-q06p krb5kdc[25723](info): set up 2 sockets
Sep 29 02:09:11 dist-test-slave-q06p krb5kdc[25723](info): commencing operation
krb5kdc: starting...
W20260929 02:09:12.079902 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:09:12 dist-test-slave-q06p krb5kdc[25723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:09:12.306439 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40123
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:40123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:12.643425 25736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:12.643700 25736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:12.643853 25736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:12.650954 25736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:09:12.651052 25736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:12.651140 25736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:12.651252 25736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:09:12.651316 25736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:09:12.661100 25736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:40123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:12.662387 25736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:12.663728 25736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:12.675239 25744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:12.675277 25741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:12.675480 25742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:12.676461 25736 server_base.cc:1034] running on GCE node
I20260929 02:09:12.677333 25736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:12.678771 25736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:12.680022 25736 hybrid_clock.cc:648] HybridClock initialized: now 1790647752679987 us; error 41 us; skew 500 ppm
I20260929 02:09:12.680385 25736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:12 dist-test-slave-q06p krb5kdc[25723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647752, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:12.683719 25736 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:09:12.685205 25736 webserver.cc:460] Webserver started at http://127.11.182.254:38413/ using document root <none> and password file <none>
I20260929 02:09:12.685972 25736 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:12.686116 25736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:12.686421 25736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:12.689370 25736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "01a0345c6d1d458182e1f44e9ffdbe04"
format_stamp: "Formatted at 2026-09-29 02:09:12 on dist-test-slave-q06p"
server_key: "d12017572ee2a51399a02487cc7a9a64"
server_key_iv: "a246f2a7ecef3fce2d9ecd9c8d297dc7"
server_key_version: "encryptionkey@0"
I20260929 02:09:12.690116 25736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "01a0345c6d1d458182e1f44e9ffdbe04"
format_stamp: "Formatted at 2026-09-29 02:09:12 on dist-test-slave-q06p"
server_key: "d12017572ee2a51399a02487cc7a9a64"
server_key_iv: "a246f2a7ecef3fce2d9ecd9c8d297dc7"
server_key_version: "encryptionkey@0"
I20260929 02:09:12.695504 25736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 02:09:12.698935 25751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:12.700239 25736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:09:12.700419 25736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "01a0345c6d1d458182e1f44e9ffdbe04"
format_stamp: "Formatted at 2026-09-29 02:09:12 on dist-test-slave-q06p"
server_key: "d12017572ee2a51399a02487cc7a9a64"
server_key_iv: "a246f2a7ecef3fce2d9ecd9c8d297dc7"
server_key_version: "encryptionkey@0"
I20260929 02:09:12.700603 25736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:12.720750 25736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:12.725010 25736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:12.725340 25736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:12.743731 25736 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:40123
I20260929 02:09:12.743753 25802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:40123 every 8 connection(s)
I20260929 02:09:12.745060 25736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:09:12.747040 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 25736
I20260929 02:09:12.747263 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:09:12.748077 11995 external_mini_cluster.cc:1428] Setting key fb0a3d7d04c88f39b38a0eade650b04e
I20260929 02:09:12.749450 25803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:12.761482 25803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04: Bootstrap starting.
Sep 29 02:09:12 dist-test-slave-q06p krb5kdc[25723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:12.764573 25803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:12.765573 25803 log.cc:824] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:12.768204 25803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04: No bootstrap required, opened a new log
I20260929 02:09:12.769615 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:12.749886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.11.182.254:40123)
0929 02:09:12.750588 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:12.750599 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:12.750637 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:09:12.752258 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:12.752278 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:12.752637 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:12.753025 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:12.754458 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:12.755805 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:12.757041 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:12.757851 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:12.758044 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:12.762751 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:12.762763 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:12.762779 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:12.762820 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:12.765898 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:12.766564 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:12.766571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:12.766577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:12.766737 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:12.767139 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:12.767144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:12.767160 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:12.767604 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:12.767892 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:12.768344 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 02:09:12.768735 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":311,"threads_started":1}
I20260929 02:09:12.773173 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:09:12.776113 25803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a0345c6d1d458182e1f44e9ffdbe04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40123 } }
I20260929 02:09:12.776507 25803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:12.776610 25803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a0345c6d1d458182e1f44e9ffdbe04, State: Initialized, Role: FOLLOWER
I20260929 02:09:12.777117 25803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a0345c6d1d458182e1f44e9ffdbe04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40123 } }
I20260929 02:09:12.777343 25803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:09:12.777513 25803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:09:12.777684 25803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:12.779810 25803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a0345c6d1d458182e1f44e9ffdbe04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40123 } }
I20260929 02:09:12.780278 25803 leader_election.cc:302] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01a0345c6d1d458182e1f44e9ffdbe04; no voters:
I20260929 02:09:12.780823 25803 leader_election.cc:288] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:09:12.780948 25808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:09:12.781643 25808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [term 1 LEADER]: Becoming Leader. State: Replica: 01a0345c6d1d458182e1f44e9ffdbe04, State: Running, Role: LEADER
I20260929 02:09:12.782153 25808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a0345c6d1d458182e1f44e9ffdbe04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40123 } }
I20260929 02:09:12.782838 25803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:09:12.785539 25809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01a0345c6d1d458182e1f44e9ffdbe04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a0345c6d1d458182e1f44e9ffdbe04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40123 } } }
I20260929 02:09:12.785312 25810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a0345c6d1d458182e1f44e9ffdbe04. Latest consensus state: current_term: 1 leader_uuid: "01a0345c6d1d458182e1f44e9ffdbe04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a0345c6d1d458182e1f44e9ffdbe04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40123 } } }
I20260929 02:09:12.786350 25809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:12.786445 25810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:12.788316 25817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:09:12.792452 25817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:09:12.799806 25817 catalog_manager.cc:1347] Generated new cluster ID: 9d65dfc71fdc43a0a4d31b7ab3a8f818
I20260929 02:09:12.799918 25817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:09:12.815552 25817 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:09:12.816810 25817 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:09:12.826061 25817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01a0345c6d1d458182e1f44e9ffdbe04: Generated new TSK 0
I20260929 02:09:12.827008 25817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:09:12.885406 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 25736
I20260929 02:09:12.886641 25798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:12.983292 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 25736
2026-09-29T02:09:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2013 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:09:13 dist-test-slave-q06p krb5kdc[25831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:09:13 dist-test-slave-q06p krb5kdc[25831](info): set up 2 sockets
Sep 29 02:09:13 dist-test-slave-q06p krb5kdc[25831](info): commencing operation
krb5kdc: starting...
W20260929 02:09:14.083736 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:09:14 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:09:14.311327 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41343
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:14.632953 25844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:14.633261 25844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:14.633351 25844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:14.640606 25844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:09:14.640724 25844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:14.640779 25844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:14.640834 25844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:09:14.640879 25844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:09:14.650974 25844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:41343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:14.652271 25844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:14.653520 25844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:14.666208 25849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:14.666250 25850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:14.666270 25852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:14.667014 25844 server_base.cc:1034] running on GCE node
I20260929 02:09:14.667829 25844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:14.669299 25844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:14.670526 25844 hybrid_clock.cc:648] HybridClock initialized: now 1790647754670470 us; error 49 us; skew 500 ppm
I20260929 02:09:14.670934 25844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:14 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:14.674113 25844 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:09:14.675802 25844 webserver.cc:460] Webserver started at http://127.11.182.254:40947/ using document root <none> and password file <none>
I20260929 02:09:14.676553 25844 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:14.676718 25844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:14.677065 25844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:14.680478 25844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "1bd51afd26cf462e838cce7450d5db04"
format_stamp: "Formatted at 2026-09-29 02:09:14 on dist-test-slave-q06p"
server_key: "cdedce427b15e9541b4520456e88e738"
server_key_iv: "f7cdf1e07b03848118c240a1cf3105ec"
server_key_version: "encryptionkey@0"
I20260929 02:09:14.681262 25844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "1bd51afd26cf462e838cce7450d5db04"
format_stamp: "Formatted at 2026-09-29 02:09:14 on dist-test-slave-q06p"
server_key: "cdedce427b15e9541b4520456e88e738"
server_key_iv: "f7cdf1e07b03848118c240a1cf3105ec"
server_key_version: "encryptionkey@0"
I20260929 02:09:14.686532 25844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:09:14.690084 25859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:14.691424 25844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:09:14.691607 25844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "1bd51afd26cf462e838cce7450d5db04"
format_stamp: "Formatted at 2026-09-29 02:09:14 on dist-test-slave-q06p"
server_key: "cdedce427b15e9541b4520456e88e738"
server_key_iv: "f7cdf1e07b03848118c240a1cf3105ec"
server_key_version: "encryptionkey@0"
I20260929 02:09:14.691800 25844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:14.706156 25844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:14.710639 25844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:14.710994 25844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:14.729480 25844 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:41343
I20260929 02:09:14.729534 25910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:41343 every 8 connection(s)
I20260929 02:09:14.730983 25844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:09:14.731979 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 25844
I20260929 02:09:14.732229 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:09:14.732862 11995 external_mini_cluster.cc:1428] Setting key e7c7e468513fc37e316f0a6f44a2cd12
I20260929 02:09:14.735487 25911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:09:14 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:14.748828 25911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Bootstrap starting.
I20260929 02:09:14.752957 25911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:14.754269 25911 log.cc:824] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:14.757356 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:14.734828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.11.182.254:41343)
0929 02:09:14.735785 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:14.735797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:14.735839 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:09:14.737470 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:14.737498 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:14.737870 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:14.738215 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:14.740068 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:14.741449 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:14.743334 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:14.743975 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:14.744114 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:14.748819 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:14.748829 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:14.748848 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:14.748896 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:14.752679 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:14.753463 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:14.753471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:14.753478 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:14.753688 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:14.754078 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:14.754085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:14.754090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:14.754808 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:14.755194 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:14.755578 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 02:09:14.756050 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":401,"threads_started":1}
I20260929 02:09:14.757848 25911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: No bootstrap required, opened a new log
I20260929 02:09:14.765981 25911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:14.766347 25911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:14.766420 25911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd51afd26cf462e838cce7450d5db04, State: Initialized, Role: FOLLOWER
I20260929 02:09:14.766968 25911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:14.767182 25911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:09:14.767279 25911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:09:14.767401 25911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:14.769580 25911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:14.770066 25911 leader_election.cc:302] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bd51afd26cf462e838cce7450d5db04; no voters:
I20260929 02:09:14.770622 25911 leader_election.cc:288] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:09:14.770766 25916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:09:14.771502 25916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd51afd26cf462e838cce7450d5db04, State: Running, Role: LEADER
I20260929 02:09:14.772091 25916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:14.772789 25911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:09:14.774712 25917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bd51afd26cf462e838cce7450d5db04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } } }
I20260929 02:09:14.774817 25918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd51afd26cf462e838cce7450d5db04. Latest consensus state: current_term: 1 leader_uuid: "1bd51afd26cf462e838cce7450d5db04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } } }
I20260929 02:09:14.775168 25917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:14.775175 25918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:14.777305 25924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:09:14.782716 25924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:09:14.791504 25924 catalog_manager.cc:1347] Generated new cluster ID: 5d7999b76f244520a8fa261034f798be
I20260929 02:09:14.791664 25924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:09:14.806707 25924 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:09:14.808063 25924 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:09:14.816900 25924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Generated new TSK 0
I20260929 02:09:14.817932 25924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:09:14.879051 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41343
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:15.205106 25937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:15.205377 25937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:15.205539 25937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:15.212695 25937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:15.212823 25937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:15.213008 25937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:09:15.222819 25937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:15.224159 25937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:15.225505 25937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:15.238323 25942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:15.238391 25943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:15.238371 25945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:15.239467 25937 server_base.cc:1034] running on GCE node
I20260929 02:09:15.240115 25937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:15.240980 25937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:15.242210 25937 hybrid_clock.cc:648] HybridClock initialized: now 1790647755242155 us; error 46 us; skew 500 ppm
I20260929 02:09:15.242516 25937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:15 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:15.245666 25937 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:09:15.247174 25937 webserver.cc:460] Webserver started at http://127.11.182.193:41399/ using document root <none> and password file <none>
I20260929 02:09:15.247845 25937 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:15.247996 25937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:15.248301 25937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:15.251200 25937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "08a958a7c37c4f3d94a22774d738219b"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "38a82988837af5a85c38bdbbcf864763"
server_key_iv: "5fa6a455fe9613cb77e277a71ebf5cd0"
server_key_version: "encryptionkey@0"
I20260929 02:09:15.251967 25937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "08a958a7c37c4f3d94a22774d738219b"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "38a82988837af5a85c38bdbbcf864763"
server_key_iv: "5fa6a455fe9613cb77e277a71ebf5cd0"
server_key_version: "encryptionkey@0"
I20260929 02:09:15.257542 25937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:09:15.261025 25952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:15.262326 25937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:09:15.262497 25937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "08a958a7c37c4f3d94a22774d738219b"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "38a82988837af5a85c38bdbbcf864763"
server_key_iv: "5fa6a455fe9613cb77e277a71ebf5cd0"
server_key_version: "encryptionkey@0"
I20260929 02:09:15.262696 25937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:15.281397 25937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:15.285678 25937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:15.285964 25937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:15.286942 25937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:15.288249 25937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:15.288322 25937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:15.288513 25937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:15.288568 25937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:15.320267 25937 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:34845
I20260929 02:09:15.320308 26064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:34845 every 8 connection(s)
I20260929 02:09:15.321542 25937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:09:15.323995 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 25937
I20260929 02:09:15.324225 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:09:15.324930 11995 external_mini_cluster.cc:1428] Setting key 128203a2a950df8276129791e5ac6d49
Sep 29 02:09:15 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:15.345135 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:15.324119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:52181 (local address 127.11.182.254:41343)
0929 02:09:15.324645 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:15.324651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:15.325724 (+ 1073us) server_negotiation.cc:407] Connection header received
0929 02:09:15.328163 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:15.328166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:15.328228 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:15.328320 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:15.330355 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:15.331096 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:15.333595 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:15.333978 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:15.334145 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:15.337971 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:15.337980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:15.337987 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:15.338030 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:15.341701 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:15.342433 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:15.342441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:15.342447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:15.342602 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:15.343061 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:15.343069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:15.343075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:15.343482 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:15.343645 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:15.344309 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 02:09:15.344627 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":281,"threads_started":1}
I20260929 02:09:15.345661 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:15.324734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:52181)
0929 02:09:15.325544 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:15.325610 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:15.327886 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:15.328586 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:15.328609 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:15.329369 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:15.330179 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:15.330207 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:15.331254 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:15.331262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:15.333434 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:15.333452 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:15.334288 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:15.334297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:15.334621 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:15.335422 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:15.335448 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:15.337742 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:15.341864 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:15.341871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:15.341883 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:15.342309 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:15.342709 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:15.342712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:15.342715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:15.342952 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:15.343612 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:15.343617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:15.343911 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 02:09:15.344136 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 02:09:15.344562 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
I20260929 02:09:15.347256 26065 heartbeater.cc:347] Connected to a master server at 127.11.182.254:41343
I20260929 02:09:15.347571 26065 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:15.348299 26065 heartbeater.cc:511] Master 127.11.182.254:41343 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:09:15.350296 25876 ts_manager.cc:194] Registered new tserver with Master: 08a958a7c37c4f3d94a22774d738219b (127.11.182.193:34845)
I20260929 02:09:15.351801 25876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:52181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:09:15.378793 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41343
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:15.702745 26071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:15.703052 26071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:15.703146 26071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:15.710332 26071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:15.710441 26071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:15.710556 26071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:09:15.720168 26071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:15.721429 26071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:15.722674 26071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:15.737107 26077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:15.737144 26079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:15.737876 26076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:15.738348 26071 server_base.cc:1034] running on GCE node
I20260929 02:09:15.739081 26071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:15.740206 26071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:15.741423 26071 hybrid_clock.cc:648] HybridClock initialized: now 1790647755741350 us; error 63 us; skew 500 ppm
I20260929 02:09:15.741757 26071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:15 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:15.745590 26071 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:09:15.747166 26071 webserver.cc:460] Webserver started at http://127.11.182.194:45795/ using document root <none> and password file <none>
I20260929 02:09:15.747903 26071 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:15.748036 26071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:15.748401 26071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:15.751331 26071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "13b79a13ce164a07b26d31a47215da6a"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "94aa4f71381d21555b943134640c38fe"
server_key_iv: "62feae85874016e2d839c9438fbf9c29"
server_key_version: "encryptionkey@0"
I20260929 02:09:15.752111 26071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "13b79a13ce164a07b26d31a47215da6a"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "94aa4f71381d21555b943134640c38fe"
server_key_iv: "62feae85874016e2d839c9438fbf9c29"
server_key_version: "encryptionkey@0"
I20260929 02:09:15.757121 26071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 02:09:15.760497 26086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:15.761745 26071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:09:15.761922 26071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "13b79a13ce164a07b26d31a47215da6a"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "94aa4f71381d21555b943134640c38fe"
server_key_iv: "62feae85874016e2d839c9438fbf9c29"
server_key_version: "encryptionkey@0"
I20260929 02:09:15.762123 26071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:15.789676 26071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:15.793465 26071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:15.793762 26071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:15.794677 26071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:15.796121 26071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:15.796242 26071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:15.796339 26071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:15.796382 26071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:15.828723 26071 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:44835
I20260929 02:09:15.828807 26198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:44835 every 8 connection(s)
I20260929 02:09:15.829959 26071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:09:15.839948 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 26071
I20260929 02:09:15.840143 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:09:15.840696 11995 external_mini_cluster.cc:1428] Setting key be80655b12370b7f71be1b1e4e2612d4
Sep 29 02:09:15 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:15.850078 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:15.832459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:56529 (local address 127.11.182.254:41343)
0929 02:09:15.832648 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:15.832652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:15.833856 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 02:09:15.835733 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:15.835737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:15.835802 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:15.835895 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:15.837301 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:15.837995 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:15.839660 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:15.840342 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:15.840471 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:15.844353 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:15.844358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:15.844362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:15.844391 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:15.847071 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:15.847953 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:15.847957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:15.847959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:15.848086 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:15.848553 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:15.848556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:15.848558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:15.848786 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:15.848889 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:15.849686 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 02:09:15.849841 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:09:15.851656 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:15.832912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:56529)
0929 02:09:15.833695 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:15.833746 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:15.835500 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:15.836117 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:15.836125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:15.836605 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:15.837159 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:15.837175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:15.838116 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:15.838119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:15.839487 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:15.839503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:15.840539 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:15.840544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:15.840729 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:15.841445 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:15.841477 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:15.844128 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:15.847200 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:15.847212 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:15.847231 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:15.847841 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:15.848194 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:15.848202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:15.848208 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:15.848440 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:15.848903 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:15.848913 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:15.849325 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 02:09:15.849607 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 02:09:15.850161 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":335,"threads_started":1}
I20260929 02:09:15.853013 26199 heartbeater.cc:347] Connected to a master server at 127.11.182.254:41343
I20260929 02:09:15.853318 26199 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:15.854144 26199 heartbeater.cc:511] Master 127.11.182.254:41343 requested a full tablet report, sending...
I20260929 02:09:15.855693 25876 ts_manager.cc:194] Registered new tserver with Master: 13b79a13ce164a07b26d31a47215da6a (127.11.182.194:44835)
I20260929 02:09:15.856467 25876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:56529
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:09:15.894008 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41343
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:16.210225 26204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:16.210480 26204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:16.210569 26204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:16.217630 26204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:16.217730 26204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:16.217839 26204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:09:16.227770 26204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:16.228926 26204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:16.230106 26204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:16.242789 26210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:16.242765 26212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:16.243291 26209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:16.243613 26204 server_base.cc:1034] running on GCE node
I20260929 02:09:16.244290 26204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:16.245232 26204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:16.246492 26204 hybrid_clock.cc:648] HybridClock initialized: now 1790647756246433 us; error 60 us; skew 500 ppm
I20260929 02:09:16.246897 26204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:16 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647756, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:16.250093 26204 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:09:16.251574 26204 webserver.cc:460] Webserver started at http://127.11.182.195:39725/ using document root <none> and password file <none>
I20260929 02:09:16.252280 26204 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:16.252440 26204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:16.252769 26204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:16.256354 26204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "18fbd776335d4a3193af4ae08c70973e"
format_stamp: "Formatted at 2026-09-29 02:09:16 on dist-test-slave-q06p"
server_key: "576ffed1c4e15ae331860bd498a81b72"
server_key_iv: "44268dc02bebb4e069719d6b23757694"
server_key_version: "encryptionkey@0"
I20260929 02:09:16.257251 26204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "18fbd776335d4a3193af4ae08c70973e"
format_stamp: "Formatted at 2026-09-29 02:09:16 on dist-test-slave-q06p"
server_key: "576ffed1c4e15ae331860bd498a81b72"
server_key_iv: "44268dc02bebb4e069719d6b23757694"
server_key_version: "encryptionkey@0"
I20260929 02:09:16.262491 26204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:09:16.266163 26219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:16.267706 26204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:09:16.267838 26204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "18fbd776335d4a3193af4ae08c70973e"
format_stamp: "Formatted at 2026-09-29 02:09:16 on dist-test-slave-q06p"
server_key: "576ffed1c4e15ae331860bd498a81b72"
server_key_iv: "44268dc02bebb4e069719d6b23757694"
server_key_version: "encryptionkey@0"
I20260929 02:09:16.267990 26204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:16.301484 26204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:16.305487 26204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:16.305730 26204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:16.306583 26204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:16.307850 26204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:16.307924 26204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:16.308012 26204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:16.308055 26204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:16.340947 26204 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:39119
I20260929 02:09:16.340991 26331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:39119 every 8 connection(s)
I20260929 02:09:16.342332 26204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:09:16.344470 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 26204
I20260929 02:09:16.344651 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:09:16.345156 11995 external_mini_cluster.cc:1428] Setting key 7d45d4fbeecb70c91bac21feb2823158
I20260929 02:09:16.354615 26065 heartbeater.cc:503] Master 127.11.182.254:41343 was elected leader, sending a full tablet report...
Sep 29 02:09:16 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647756, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:16.362360 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.345189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:38589 (local address 127.11.182.254:41343)
0929 02:09:16.345415 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.345423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.346970 (+ 1547us) server_negotiation.cc:407] Connection header received
0929 02:09:16.348905 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.348910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.349034 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.349141 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:16.350533 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.351345 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.352845 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.353240 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.353378 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.356798 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:16.356803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:16.356806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:16.356837 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:16.359484 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:16.360154 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:16.360161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:16.360165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:16.360301 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:16.360727 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:16.360733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:16.360736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:16.361000 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:16.361151 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.361795 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.362019 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 02:09:16.363266 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.345777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:38589)
0929 02:09:16.346779 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:16.346820 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:16.348639 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:16.349362 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:16.349370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:16.349848 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:16.350371 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.350387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.351502 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.351506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.352697 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.352709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.353474 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.353481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.353708 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.354361 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:16.354387 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:16.356608 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:16.359620 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:16.359628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:16.359642 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:16.360028 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:16.360419 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:16.360424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:16.360428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:16.360613 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:16.361121 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:16.361129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:16.361437 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 02:09:16.361695 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 02:09:16.362171 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":522,"threads_started":1}
I20260929 02:09:16.364609 26332 heartbeater.cc:347] Connected to a master server at 127.11.182.254:41343
I20260929 02:09:16.364861 26332 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:16.365661 26332 heartbeater.cc:511] Master 127.11.182.254:41343 requested a full tablet report, sending...
I20260929 02:09:16.367106 25876 ts_manager.cc:194] Registered new tserver with Master: 18fbd776335d4a3193af4ae08c70973e (127.11.182.195:39119)
I20260929 02:09:16.367830 25876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:38589
I20260929 02:09:16.373000 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:09:16.389727 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.377677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.11.182.254:41343)
0929 02:09:16.377905 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.377915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.378255 (+ 340us) server_negotiation.cc:407] Connection header received
0929 02:09:16.378411 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.378419 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.378487 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.378574 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:16.379972 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.380683 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.381953 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.382347 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.382435 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.384612 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:16.384617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:16.384620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:16.384648 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:16.387315 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:16.387905 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:16.387909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:16.387911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:16.388031 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:16.388349 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:16.388352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:16.388354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:16.388676 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:16.388853 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.389218 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.389369 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:09:16.393698 25876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:09:16.400609 25876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:09:16.423431 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.412870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.11.182.193:34845)
0929 02:09:16.414649 (+ 1779us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.414663 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.414704 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:09:16.414836 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.414870 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.415189 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.415474 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:16.416337 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.418017 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.420806 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.422120 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.422324 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.422506 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.422917 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.423084 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1476,"thread_start_us":1281,"threads_started":1}
I20260929 02:09:16.423623 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.412519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:34845 (local address 127.0.0.1:60556)
0929 02:09:16.413205 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:16.413251 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:16.413399 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:16.415486 (+ 2087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:16.415498 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:16.415563 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:16.416068 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.416085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.418177 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.418182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.420625 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.420640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.422369 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.422376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.422625 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.422682 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 02:09:16.422837 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 02:09:16.423087 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":4,"thread_start_us":331,"threads_started":1}
I20260929 02:09:16.425658 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.413754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:39119 (local address 127.0.0.1:54768)
0929 02:09:16.416604 (+ 2850us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:16.416620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:16.416775 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:16.417768 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:16.417775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:16.417832 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:16.418728 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.418742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.420808 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.420814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.422915 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.422939 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.424827 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.424834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.425003 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.425051 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 02:09:16.425132 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:09:16.425281 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2731,"thread_start_us":2580,"threads_started":1}
I20260929 02:09:16.426481 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.414085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.11.182.195:39119)
0929 02:09:16.416351 (+ 2266us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.416362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.416912 (+ 550us) server_negotiation.cc:407] Connection header received
0929 02:09:16.417056 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.417063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.417227 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.417443 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:16.418992 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.420612 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.423109 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.424612 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.424827 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.425008 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.425927 (+ 919us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.426099 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2080,"thread_start_us":1924,"threads_started":1}
I20260929 02:09:16.427467 26000 tablet_service.cc:1467] Processing CreateTablet for tablet fd80e8bd46184009bdcbec1c69fcc874 (DEFAULT_TABLE table=test-table [id=4738e2b60d684673853f0c04c1e59b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:16.428766 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.417114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.11.182.194:44835)
0929 02:09:16.417740 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.417749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.418826 (+ 1077us) server_negotiation.cc:407] Connection header received
0929 02:09:16.419133 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.419143 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.419374 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.419591 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:16.420226 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.421751 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.425141 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.427455 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.427662 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.427833 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.428071 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.428435 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":324,"threads_started":1}
I20260929 02:09:16.429036 26000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd80e8bd46184009bdcbec1c69fcc874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:16.429709 26349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.417700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:44835 (local address 127.0.0.1:36514)
0929 02:09:16.418672 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:16.418708 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:16.418864 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:16.419600 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:16.419606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:16.419640 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:16.420065 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.420076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.421906 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.421911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.424420 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.424433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.427683 (+ 3250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.427689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.427851 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.427883 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:09:16.429278 (+ 1395us) client_negotiation.cc:240] Negotiation successful
0929 02:09:16.429413 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":288,"threads_started":1}
I20260929 02:09:16.430027 26267 tablet_service.cc:1467] Processing CreateTablet for tablet fd80e8bd46184009bdcbec1c69fcc874 (DEFAULT_TABLE table=test-table [id=4738e2b60d684673853f0c04c1e59b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:16.431267 26267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd80e8bd46184009bdcbec1c69fcc874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:16.432976 26134 tablet_service.cc:1467] Processing CreateTablet for tablet fd80e8bd46184009bdcbec1c69fcc874 (DEFAULT_TABLE table=test-table [id=4738e2b60d684673853f0c04c1e59b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:16.434628 26134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd80e8bd46184009bdcbec1c69fcc874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:16.442874 26350 tablet_bootstrap.cc:492] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: Bootstrap starting.
I20260929 02:09:16.446091 26350 tablet_bootstrap.cc:654] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:16.447502 26350 log.cc:824] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:16.448102 26351 tablet_bootstrap.cc:492] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: Bootstrap starting.
I20260929 02:09:16.449906 26353 tablet_bootstrap.cc:492] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: Bootstrap starting.
I20260929 02:09:16.451404 26350 tablet_bootstrap.cc:492] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: No bootstrap required, opened a new log
I20260929 02:09:16.451730 26350 ts_tablet_manager.cc:1397] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:09:16.451920 26351 tablet_bootstrap.cc:654] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:16.453167 26351 log.cc:824] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:16.453378 26353 tablet_bootstrap.cc:654] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:16.454420 26353 log.cc:824] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:16.456921 26351 tablet_bootstrap.cc:492] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: No bootstrap required, opened a new log
I20260929 02:09:16.457202 26351 ts_tablet_manager.cc:1397] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:09:16.457314 26353 tablet_bootstrap.cc:492] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: No bootstrap required, opened a new log
I20260929 02:09:16.457561 26353 ts_tablet_manager.cc:1397] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 02:09:16.459983 26350 raft_consensus.cc:348] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:16.460328 26350 raft_consensus.cc:374] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:16.460455 26350 raft_consensus.cc:729] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18fbd776335d4a3193af4ae08c70973e, State: Initialized, Role: FOLLOWER
I20260929 02:09:16.461031 26350 consensus_queue.cc:260] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:16.462105 26332 heartbeater.cc:503] Master 127.11.182.254:41343 was elected leader, sending a full tablet report...
I20260929 02:09:16.463218 26350 ts_tablet_manager.cc:1428] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260929 02:09:16.468024 26353 raft_consensus.cc:348] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:16.468024 26351 raft_consensus.cc:348] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:16.468453 26351 raft_consensus.cc:374] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:16.468458 26353 raft_consensus.cc:374] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:16.468557 26351 raft_consensus.cc:729] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a958a7c37c4f3d94a22774d738219b, State: Initialized, Role: FOLLOWER
I20260929 02:09:16.468556 26353 raft_consensus.cc:729] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b79a13ce164a07b26d31a47215da6a, State: Initialized, Role: FOLLOWER
I20260929 02:09:16.469148 26353 consensus_queue.cc:260] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:16.469161 26351 consensus_queue.cc:260] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:16.470201 26199 heartbeater.cc:503] Master 127.11.182.254:41343 was elected leader, sending a full tablet report...
I20260929 02:09:16.471002 26353 ts_tablet_manager.cc:1428] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260929 02:09:16.471243 26351 ts_tablet_manager.cc:1428] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
W20260929 02:09:16.573918 26066 tablet.cc:2369] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:09:16.581933 26200 tablet.cc:2369] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:09:16.594208 26333 tablet.cc:2369] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:09:16.649966 26356 raft_consensus.cc:482] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:09:16.650295 26356 raft_consensus.cc:504] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:16.652115 26356 leader_election.cc:288] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08a958a7c37c4f3d94a22774d738219b (127.11.182.193:34845), 13b79a13ce164a07b26d31a47215da6a (127.11.182.194:44835)
I20260929 02:09:16.657646 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.652469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:34845 (local address 127.11.182.195:55913)
0929 02:09:16.652693 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:16.652714 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:16.652824 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:16.653374 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:16.653378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:16.653416 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:16.653576 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.653583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.654565 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.654570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.655866 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.655878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.656942 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.656948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.657127 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.657159 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:09:16.657259 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:09:16.657388 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:09:16.657662 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.652583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:55913 (local address 127.11.182.193:34845)
0929 02:09:16.652796 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.652802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.652823 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:09:16.653004 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.653012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.653089 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.653192 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:16.653719 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.654418 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.656028 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.656792 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.656960 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.657032 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.657293 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.657412 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:09:16.658638 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.652889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:44835 (local address 127.11.182.195:34747)
0929 02:09:16.653667 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:16.653681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:16.653822 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:16.654146 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:16.654150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:16.654173 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:16.654488 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.654496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.655559 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.655564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.656868 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:16.656878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.658038 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.658044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:16.658233 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.658271 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:09:16.658342 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 02:09:16.658421 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":474,"threads_started":1}
I20260929 02:09:16.658811 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.653095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:34747 (local address 127.11.182.194:44835)
0929 02:09:16.653311 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.653319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.653776 (+ 457us) server_negotiation.cc:407] Connection header received
0929 02:09:16.653973 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.653977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.654035 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.654134 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:16.654646 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.655404 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.657028 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.657852 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.657996 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.658072 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.658401 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.658547 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:09:16.658771 26020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80e8bd46184009bdcbec1c69fcc874" candidate_uuid: "18fbd776335d4a3193af4ae08c70973e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a958a7c37c4f3d94a22774d738219b" is_pre_election: true
I20260929 02:09:16.659480 26020 raft_consensus.cc:2454] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18fbd776335d4a3193af4ae08c70973e in term 0.
I20260929 02:09:16.659828 26154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80e8bd46184009bdcbec1c69fcc874" candidate_uuid: "18fbd776335d4a3193af4ae08c70973e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b79a13ce164a07b26d31a47215da6a" is_pre_election: true
I20260929 02:09:16.660203 26220 leader_election.cc:302] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a958a7c37c4f3d94a22774d738219b, 18fbd776335d4a3193af4ae08c70973e; no voters:
I20260929 02:09:16.660315 26154 raft_consensus.cc:2454] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18fbd776335d4a3193af4ae08c70973e in term 0.
I20260929 02:09:16.660566 26356 raft_consensus.cc:2781] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:09:16.660702 26356 raft_consensus.cc:482] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:09:16.660838 26356 raft_consensus.cc:3037] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:16.663105 26356 raft_consensus.cc:504] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:16.663749 26356 leader_election.cc:288] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [CANDIDATE]: Term 1 election: Requested vote from peers 08a958a7c37c4f3d94a22774d738219b (127.11.182.193:34845), 13b79a13ce164a07b26d31a47215da6a (127.11.182.194:44835)
I20260929 02:09:16.664147 26020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80e8bd46184009bdcbec1c69fcc874" candidate_uuid: "18fbd776335d4a3193af4ae08c70973e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a958a7c37c4f3d94a22774d738219b"
I20260929 02:09:16.664191 26154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd80e8bd46184009bdcbec1c69fcc874" candidate_uuid: "18fbd776335d4a3193af4ae08c70973e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b79a13ce164a07b26d31a47215da6a"
I20260929 02:09:16.664367 26154 raft_consensus.cc:3037] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:16.664368 26020 raft_consensus.cc:3037] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:16.667656 26154 raft_consensus.cc:2454] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18fbd776335d4a3193af4ae08c70973e in term 1.
I20260929 02:09:16.667656 26020 raft_consensus.cc:2454] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18fbd776335d4a3193af4ae08c70973e in term 1.
I20260929 02:09:16.668017 26220 leader_election.cc:302] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b79a13ce164a07b26d31a47215da6a, 18fbd776335d4a3193af4ae08c70973e; no voters:
I20260929 02:09:16.668304 26356 raft_consensus.cc:2781] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:09:16.668735 26356 raft_consensus.cc:686] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 1 LEADER]: Becoming Leader. State: Replica: 18fbd776335d4a3193af4ae08c70973e, State: Running, Role: LEADER
I20260929 02:09:16.669178 26356 consensus_queue.cc:237] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:16.673689 25876 catalog_manager.cc:5495] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e reported cstate change: term changed from 0 to 1, leader changed from <none> to 18fbd776335d4a3193af4ae08c70973e (127.11.182.195). New cstate: current_term: 1 leader_uuid: "18fbd776335d4a3193af4ae08c70973e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } health_report { overall_health: UNKNOWN } } }
I20260929 02:09:16.721468 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.716284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.11.182.195:39119)
0929 02:09:16.716492 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.716498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.716580 (+ 82us) server_negotiation.cc:407] Connection header received
0929 02:09:16.716768 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.716772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.716847 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.716967 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:09:16.717317 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.718047 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.719630 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.720005 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.720093 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.720548 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:09:16.721006 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:09:16.721115 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.721207 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.721269 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:09:16.729799 26154 raft_consensus.cc:1261] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [term 1 FOLLOWER]: Refusing update from remote peer 18fbd776335d4a3193af4ae08c70973e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:16.729908 26020 raft_consensus.cc:1261] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [term 1 FOLLOWER]: Refusing update from remote peer 18fbd776335d4a3193af4ae08c70973e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:16.730861 26360 consensus_queue.cc:1035] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:16.731313 26356 consensus_queue.cc:1035] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:16.745082 26363 mvcc.cc:205] Tried to move back new op lower bound from 7334493211554770944 to 7334493211320483840. Current Snapshot: MvccSnapshot[applied={T|T < 7334493211554770944}]
I20260929 02:09:16.747476 26365 mvcc.cc:205] Tried to move back new op lower bound from 7334493211554770944 to 7334493211320483840. Current Snapshot: MvccSnapshot[applied={T|T < 7334493211554770944}]
I20260929 02:09:16.761617 25875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:09:16.761929 25875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:09:16.766095 25875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Sending DeleteTablet for 3 replicas of tablet fd80e8bd46184009bdcbec1c69fcc874
I20260929 02:09:16.767756 26267 tablet_service.cc:1514] Processing DeleteTablet for tablet fd80e8bd46184009bdcbec1c69fcc874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:16 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:54768
I20260929 02:09:16.768117 26134 tablet_service.cc:1514] Processing DeleteTablet for tablet fd80e8bd46184009bdcbec1c69fcc874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:16 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:36514
I20260929 02:09:16.768529 26000 tablet_service.cc:1514] Processing DeleteTablet for tablet fd80e8bd46184009bdcbec1c69fcc874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:16 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:60556
I20260929 02:09:16.768795 26372 tablet_replica.cc:330] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: stopping tablet replica
I20260929 02:09:16.769398 26372 raft_consensus.cc:2229] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:09:16.769567 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 25937
I20260929 02:09:16.770358 26372 raft_consensus.cc:2258] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:16.777343 26372 ts_tablet_manager.cc:1905] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:16.784096 26373 tablet_replica.cc:330] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: stopping tablet replica
I20260929 02:09:16.784705 26373 raft_consensus.cc:2229] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:09:16.785310 26373 raft_consensus.cc:2258] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:16.786298 26372 ts_tablet_manager.cc:1918] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:16.786484 26372 log.cc:1196] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/fd80e8bd46184009bdcbec1c69fcc874
I20260929 02:09:16.787531 26374 tablet_replica.cc:330] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: stopping tablet replica
I20260929 02:09:16.787940 26373 ts_tablet_manager.cc:1905] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:16.787925 26374 raft_consensus.cc:2229] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:09:16.788419 26374 raft_consensus.cc:2258] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:16.790285 26374 ts_tablet_manager.cc:1905] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:16.794656 26373 ts_tablet_manager.cc:1918] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:16.794879 26373 log.cc:1196] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/fd80e8bd46184009bdcbec1c69fcc874
Sep 29 02:09:16 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:09:16.795506 26373 ts_tablet_manager.cc:1939] T fd80e8bd46184009bdcbec1c69fcc874 P 13b79a13ce164a07b26d31a47215da6a: Deleting consensus metadata
I20260929 02:09:16.796484 26372 ts_tablet_manager.cc:1939] T fd80e8bd46184009bdcbec1c69fcc874 P 18fbd776335d4a3193af4ae08c70973e: Deleting consensus metadata
I20260929 02:09:16.797447 25860 catalog_manager.cc:4841] TS 13b79a13ce164a07b26d31a47215da6a (127.11.182.194:44835): tablet fd80e8bd46184009bdcbec1c69fcc874 (table test-table [id=4738e2b60d684673853f0c04c1e59b96]) successfully deleted
I20260929 02:09:16.798157 25860 catalog_manager.cc:4841] TS 18fbd776335d4a3193af4ae08c70973e (127.11.182.195:39119): tablet fd80e8bd46184009bdcbec1c69fcc874 (table test-table [id=4738e2b60d684673853f0c04c1e59b96]) successfully deleted
I20260929 02:09:16.799149 26374 ts_tablet_manager.cc:1918] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:16.799317 26374 log.cc:1196] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/fd80e8bd46184009bdcbec1c69fcc874
I20260929 02:09:16.799793 26374 ts_tablet_manager.cc:1939] T fd80e8bd46184009bdcbec1c69fcc874 P 08a958a7c37c4f3d94a22774d738219b: Deleting consensus metadata
I20260929 02:09:16.801126 25860 catalog_manager.cc:4841] TS 08a958a7c37c4f3d94a22774d738219b (127.11.182.193:34845): tablet fd80e8bd46184009bdcbec1c69fcc874 (table test-table [id=4738e2b60d684673853f0c04c1e59b96]) successfully deleted
I20260929 02:09:16.803681 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.770776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.11.182.193:34845)
0929 02:09:16.770981 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.770988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.776648 (+ 5660us) server_negotiation.cc:407] Connection header received
0929 02:09:16.776857 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.776863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.776952 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.777078 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:16.786622 (+ 9544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.787934 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.790277 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.790964 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.791067 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.796836 (+ 5769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:16.796849 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:16.796870 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:16.796922 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:16.800875 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:16.801729 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:16.801736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:16.801738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:16.801863 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:16.802306 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:16.802310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:16.802312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:16.802649 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:16.802912 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.803231 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.803387 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:09:16.804415 26060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:16.894569 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 25937
I20260929 02:09:16.912232 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 26071
Sep 29 02:09:16 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:09:16.929373 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:16.913211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.11.182.194:44835)
0929 02:09:16.913426 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:16.913433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:16.913454 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:09:16.913523 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:16.913529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:16.913622 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:16.913742 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:16.915010 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.916189 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.917514 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:16.918152 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:16.918283 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:16.922574 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:16.922593 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:16.922614 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:16.922658 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:16.926501 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:16.927201 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:16.927212 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:16.927216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:16.927367 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:16.927709 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:16.927715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:16.927718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:16.928253 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:16.928540 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:16.928755 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 02:09:16.928973 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:09:16.930222 26194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:17.019434 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 26071
I20260929 02:09:17.036814 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 26204
Sep 29 02:09:17 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:09:17.055091 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:17.037646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.11.182.195:39119)
0929 02:09:17.037865 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:17.037872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:17.037903 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:09:17.038094 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:17.038100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:17.038182 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:17.038277 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:17.040141 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:17.040928 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:17.042906 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:17.043389 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:17.043524 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:17.048946 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:17.048958 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:17.048978 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:17.049021 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:17.052243 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:17.052917 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:17.052933 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:17.052936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:17.053081 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:17.053460 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:17.053472 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:17.053474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:17.053842 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:17.054146 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:17.054442 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 02:09:17.054681 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:09:17.055716 26327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:17.176666 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 26204
I20260929 02:09:17.195745 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 25844
I20260929 02:09:17.196902 25906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:17.306207 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 25844
I20260929 02:09:17.327306 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41343
--webserver_interface=127.11.182.254
--webserver_port=40947
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:17.649124 26380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:17.649416 26380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:17.649576 26380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:17.656726 26380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:09:17.656818 26380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:17.656895 26380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:17.656984 26380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:09:17.657071 26380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:09:17.667019 26380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:41343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=40947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:17.668322 26380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:17.669618 26380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:17.691654 26386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:17.691605 26389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:17.691597 26387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:17.692694 26380 server_base.cc:1034] running on GCE node
I20260929 02:09:17.693569 26380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:17.695160 26380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:17.696393 26380 hybrid_clock.cc:648] HybridClock initialized: now 1790647757696344 us; error 50 us; skew 500 ppm
I20260929 02:09:17.696789 26380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:17 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647757, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:17.700309 26380 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:09:17.701920 26380 webserver.cc:460] Webserver started at http://127.11.182.254:40947/ using document root <none> and password file <none>
I20260929 02:09:17.702684 26380 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:17.702884 26380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:17.708763 26380 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 02:09:17.711701 26396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:17.713011 26380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:09:17.713189 26380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "1bd51afd26cf462e838cce7450d5db04"
format_stamp: "Formatted at 2026-09-29 02:09:14 on dist-test-slave-q06p"
server_key: "cdedce427b15e9541b4520456e88e738"
server_key_iv: "f7cdf1e07b03848118c240a1cf3105ec"
server_key_version: "encryptionkey@0"
I20260929 02:09:17.714238 26380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:17.729816 26380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:17.733863 26380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:17.734134 26380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:17.753309 26380 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:41343
I20260929 02:09:17.753329 26447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:41343 every 8 connection(s)
I20260929 02:09:17.754833 26380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:09:17.758643 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 26380
I20260929 02:09:17.759493 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:34845
--local_ip_for_outbound_sockets=127.11.182.193
--tserver_master_addrs=127.11.182.254:41343
--webserver_port=41399
--webserver_interface=127.11.182.193
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:09:17.762007 26448 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:09:17.765331 26448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Bootstrap starting.
I20260929 02:09:17.794039 26448 log.cc:824] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:17.809456 26448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:09:17.810098 26448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Bootstrap complete.
I20260929 02:09:17.824463 26448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:17.826011 26448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd51afd26cf462e838cce7450d5db04, State: Initialized, Role: FOLLOWER
I20260929 02:09:17.826877 26448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:17.827237 26448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:09:17.827399 26448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:09:17.827606 26448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:09:17.833344 26448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:17.833994 26448 leader_election.cc:302] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bd51afd26cf462e838cce7450d5db04; no voters:
I20260929 02:09:17.834595 26448 leader_election.cc:288] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:09:17.834734 26452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:09:17.836174 26452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 2 LEADER]: Becoming Leader. State: Replica: 1bd51afd26cf462e838cce7450d5db04, State: Running, Role: LEADER
I20260929 02:09:17.836728 26448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:09:17.836679 26452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:17.839386 26453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1bd51afd26cf462e838cce7450d5db04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } } }
I20260929 02:09:17.839682 26453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:17.839035 26454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd51afd26cf462e838cce7450d5db04. Latest consensus state: current_term: 2 leader_uuid: "1bd51afd26cf462e838cce7450d5db04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } } }
I20260929 02:09:17.840216 26454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:17.848641 26459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:09:17.856439 26459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:09:17.857810 26459 catalog_manager.cc:1259] Loaded cluster ID: 5d7999b76f244520a8fa261034f798be
I20260929 02:09:17.857961 26459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:09:17.861176 26459 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 02:09:17.880059 26459 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 02:09:17.888861 26459 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:09:17.890185 26459 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Loaded TSK: 0
I20260929 02:09:17.891418 26459 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:09:18.150421 26450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:18.150743 26450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:18.150942 26450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:18.158288 26450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:18.158443 26450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:18.158629 26450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:09:18.168672 26450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:34845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=41399
--tserver_master_addrs=127.11.182.254:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:18.170077 26450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:18.171509 26450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:18.184944 26476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:18.185738 26475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:18.185803 26478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:18.186301 26450 server_base.cc:1034] running on GCE node
I20260929 02:09:18.187036 26450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:18.188062 26450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:18.189281 26450 hybrid_clock.cc:648] HybridClock initialized: now 1790647758189226 us; error 50 us; skew 500 ppm
I20260929 02:09:18.189612 26450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:18 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:18.199168 26450 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:09:18.200851 26450 webserver.cc:460] Webserver started at http://127.11.182.193:41399/ using document root <none> and password file <none>
I20260929 02:09:18.201565 26450 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:18.201712 26450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:18.207190 26450 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:09:18.209911 26485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:18.211344 26450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:09:18.211505 26450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "08a958a7c37c4f3d94a22774d738219b"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "38a82988837af5a85c38bdbbcf864763"
server_key_iv: "5fa6a455fe9613cb77e277a71ebf5cd0"
server_key_version: "encryptionkey@0"
I20260929 02:09:18.212445 26450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:18.238469 26450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:18.242949 26450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:18.243234 26450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:18.244235 26450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:18.245648 26450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:18.245733 26450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:18.245879 26450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:18.245960 26450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:18.280282 26450 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:34845
I20260929 02:09:18.280381 26597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:34845 every 8 connection(s)
I20260929 02:09:18.281772 26450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:09:18.286341 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 26450
I20260929 02:09:18.287171 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:44835
--local_ip_for_outbound_sockets=127.11.182.194
--tserver_master_addrs=127.11.182.254:41343
--webserver_port=45795
--webserver_interface=127.11.182.194
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:09:18 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:18.305693 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:18.284708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:57167 (local address 127.11.182.254:41343)
0929 02:09:18.285320 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:18.285332 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:18.286933 (+ 1601us) server_negotiation.cc:407] Connection header received
0929 02:09:18.288473 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:18.288485 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:18.288819 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:18.289185 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:18.290681 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:18.291974 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:18.293506 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:18.294147 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:18.294339 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:18.298135 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:18.298143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:18.298157 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:18.298193 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:18.301082 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:18.301850 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:18.301856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:18.301862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:18.302019 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:18.302470 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:18.302474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:18.302476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:18.303057 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:18.303260 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:18.304020 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 02:09:18.304405 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":267,"threads_started":1}
I20260929 02:09:18.305717 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:18.284944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:57167)
0929 02:09:18.285869 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:18.285925 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:18.287925 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:18.289310 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:18.289322 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:18.289920 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:18.290448 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:18.290466 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:18.292142 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:18.292149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:18.293361 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:18.293373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:18.294394 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:18.294402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:18.294663 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:18.295435 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:18.295458 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:18.297846 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:18.301240 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:18.301249 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:18.301264 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:18.301698 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:18.302137 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:18.302143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:18.302147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:18.302335 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:18.303214 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:18.303223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:18.303546 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 02:09:18.303821 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 02:09:18.304381 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":397,"threads_started":1}
I20260929 02:09:18.308665 26598 heartbeater.cc:347] Connected to a master server at 127.11.182.254:41343
I20260929 02:09:18.309064 26598 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:18.310103 26598 heartbeater.cc:511] Master 127.11.182.254:41343 requested a full tablet report, sending...
I20260929 02:09:18.312395 26413 ts_manager.cc:194] Registered new tserver with Master: 08a958a7c37c4f3d94a22774d738219b (127.11.182.193:34845)
I20260929 02:09:18.313835 26413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:57167
W20260929 02:09:18.619359 26602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:18.619632 26602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:18.619803 26602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:18.626946 26602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:18.627074 26602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:18.627280 26602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:09:18.636865 26602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:44835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=45795
--tserver_master_addrs=127.11.182.254:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:18.638161 26602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:18.639467 26602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:18.653016 26607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:18.653082 26608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:18.653085 26610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:18.653956 26602 server_base.cc:1034] running on GCE node
I20260929 02:09:18.654574 26602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:18.655555 26602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:18.656785 26602 hybrid_clock.cc:648] HybridClock initialized: now 1790647758656731 us; error 66 us; skew 500 ppm
I20260929 02:09:18.657122 26602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:18 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:18.660281 26602 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:09:18.661645 26602 webserver.cc:460] Webserver started at http://127.11.182.194:45795/ using document root <none> and password file <none>
I20260929 02:09:18.662341 26602 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:18.662534 26602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:18.667867 26602 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:09:18.670578 26617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:18.672133 26602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:09:18.672358 26602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "13b79a13ce164a07b26d31a47215da6a"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "94aa4f71381d21555b943134640c38fe"
server_key_iv: "62feae85874016e2d839c9438fbf9c29"
server_key_version: "encryptionkey@0"
I20260929 02:09:18.673453 26602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:18.686231 26602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:18.691118 26602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:18.691421 26602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:18.692441 26602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:18.693835 26602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:18.693971 26602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:18.694123 26602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:18.694222 26602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:18.729681 26602 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:44835
I20260929 02:09:18.729740 26729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:44835 every 8 connection(s)
I20260929 02:09:18.731333 26602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:09:18.740651 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 26602
I20260929 02:09:18.741276 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:39119
--local_ip_for_outbound_sockets=127.11.182.195
--tserver_master_addrs=127.11.182.254:41343
--webserver_port=39725
--webserver_interface=127.11.182.195
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:09:18 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:18.751259 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:18.733866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:33583 (local address 127.11.182.254:41343)
0929 02:09:18.734039 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:18.734043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:18.735244 (+ 1201us) server_negotiation.cc:407] Connection header received
0929 02:09:18.737125 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:18.737131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:18.737226 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:18.737339 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:18.738829 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:18.739719 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:18.741311 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:18.741869 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:18.742004 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:18.745378 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:18.745383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:18.745387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:18.745418 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:18.748016 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:18.748741 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:18.748749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:18.748757 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:18.748877 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:18.749309 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:18.749316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:18.749320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:18.749614 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:18.749772 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:18.750787 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0929 02:09:18.751020 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:09:18.753021 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:18.734320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:33583)
0929 02:09:18.735077 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:18.735121 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:18.736848 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:18.737593 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:18.737601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:18.738082 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:18.738646 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:18.738663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:18.739865 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:18.739877 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:18.741156 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:18.741167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:18.742088 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:18.742095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:18.742326 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:18.743063 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:18.743084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:18.745169 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:18.748182 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:18.748191 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:18.748207 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:18.748615 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:18.749006 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:18.749010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:18.749014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:18.749199 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:18.749818 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:18.749831 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:18.750365 (+ 534us) client_negotiation.cc:769] Sending connection context
0929 02:09:18.750719 (+ 354us) client_negotiation.cc:240] Negotiation successful
0929 02:09:18.751382 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":291,"threads_started":1}
I20260929 02:09:18.759853 26730 heartbeater.cc:347] Connected to a master server at 127.11.182.254:41343
I20260929 02:09:18.760260 26730 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:18.761286 26730 heartbeater.cc:511] Master 127.11.182.254:41343 requested a full tablet report, sending...
I20260929 02:09:18.763132 26413 ts_manager.cc:194] Registered new tserver with Master: 13b79a13ce164a07b26d31a47215da6a (127.11.182.194:44835)
I20260929 02:09:18.764222 26413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:33583
W20260929 02:09:19.074096 26733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:19.074465 26733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:19.074622 26733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:19.082165 26733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:19.082310 26733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:19.082489 26733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:09:19.092559 26733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:39119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=39725
--tserver_master_addrs=127.11.182.254:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:19.093847 26733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:19.095263 26733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:19.108108 26739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:19.108201 26738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:19.108179 26741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:19.109228 26733 server_base.cc:1034] running on GCE node
I20260929 02:09:19.109802 26733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:19.110697 26733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:19.111919 26733 hybrid_clock.cc:648] HybridClock initialized: now 1790647759111877 us; error 57 us; skew 500 ppm
I20260929 02:09:19.112262 26733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:19 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:19.115445 26733 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:09:19.116955 26733 webserver.cc:460] Webserver started at http://127.11.182.195:39725/ using document root <none> and password file <none>
I20260929 02:09:19.117668 26733 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:19.117842 26733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:19.124310 26733 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 02:09:19.127146 26748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:19.128604 26733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:09:19.128859 26733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "18fbd776335d4a3193af4ae08c70973e"
format_stamp: "Formatted at 2026-09-29 02:09:16 on dist-test-slave-q06p"
server_key: "576ffed1c4e15ae331860bd498a81b72"
server_key_iv: "44268dc02bebb4e069719d6b23757694"
server_key_version: "encryptionkey@0"
I20260929 02:09:19.129887 26733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:19.146651 26733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:19.151057 26733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:19.151326 26733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:19.152292 26733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:19.153605 26733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:19.153728 26733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:19.153864 26733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:19.153959 26733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:19.187517 26733 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:39119
I20260929 02:09:19.187603 26860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:39119 every 8 connection(s)
I20260929 02:09:19.188999 26733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:09:19.196501 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 26733
Sep 29 02:09:19 dist-test-slave-q06p krb5kdc[25831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:19.209928 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.191587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:38843 (local address 127.11.182.254:41343)
0929 02:09:19.191765 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.191769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.192920 (+ 1151us) server_negotiation.cc:407] Connection header received
0929 02:09:19.194628 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.194633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.194703 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.194792 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:19.196592 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.197677 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.199595 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.200018 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.200110 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.203847 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:19.203858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:19.203861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:19.203893 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:19.206765 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.207519 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.207524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.207528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.207669 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.208114 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.208119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.208122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.208411 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:19.208611 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.209316 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.209577 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:09:19.210958 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.192038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:38843)
0929 02:09:19.192748 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:19.192790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:19.194376 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:19.195067 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:19.195076 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:19.195592 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:19.196377 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.196405 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.197903 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.197909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.199481 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.199496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.200248 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.200257 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.200603 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.201328 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:19.201361 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:19.203625 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:19.206922 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:19.206931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:19.206949 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:19.207383 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:19.207787 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:19.207791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:19.207795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:19.207987 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:19.208570 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:19.208578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:19.208934 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 02:09:19.209205 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 02:09:19.209759 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":273,"threads_started":1}
I20260929 02:09:19.212728 26861 heartbeater.cc:347] Connected to a master server at 127.11.182.254:41343
I20260929 02:09:19.212841 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.197510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.11.182.254:41343)
0929 02:09:19.199245 (+ 1735us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.199249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.199264 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:09:19.199303 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.199307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.199360 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.199445 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:19.201386 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.202114 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.203587 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.204078 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.204225 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.207244 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:19.207249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:19.207252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:19.207281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:19.209933 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.210732 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.210740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.210745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.210941 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.211390 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.211394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.211396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.211761 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:19.211992 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.212326 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.212494 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1646,"thread_start_us":1517,"threads_started":1}
I20260929 02:09:19.213091 26861 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:19.213769 26861 heartbeater.cc:511] Master 127.11.182.254:41343 requested a full tablet report, sending...
I20260929 02:09:19.215361 26412 ts_manager.cc:194] Registered new tserver with Master: 18fbd776335d4a3193af4ae08c70973e (127.11.182.195:39119)
I20260929 02:09:19.216264 26412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:38843
I20260929 02:09:19.227977 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:09:19.235289 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.228979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.11.182.254:41343)
0929 02:09:19.229242 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.229250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.229541 (+ 291us) server_negotiation.cc:407] Connection header received
0929 02:09:19.229795 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.229807 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.229962 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.230128 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:09:19.230622 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.231411 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.232917 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.233364 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.233519 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.234128 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:09:19.234663 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:09:19.234884 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.234935 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.235018 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 02:09:19.236227 26412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:09:19.243515 26412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:09:19.265628 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.255404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.11.182.193:34845)
0929 02:09:19.256665 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.256674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.256702 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:09:19.256800 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.256808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.257075 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.257295 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:19.258305 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.259921 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.262869 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.264412 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.264580 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.264755 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.265094 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.265294 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":1011,"threads_started":1}
I20260929 02:09:19.265631 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.255195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:34845 (local address 127.0.0.1:60578)
0929 02:09:19.256276 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:19.256328 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:19.256483 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:19.257506 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:19.257516 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:19.257556 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:19.258076 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.258090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.260103 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.260112 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.262635 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.262648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.264627 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.264633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.264832 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.264882 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 02:09:19.265034 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 02:09:19.265225 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":48,"thread_start_us":720,"threads_started":1}
I20260929 02:09:19.267170 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.256691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:39119 (local address 127.0.0.1:54792)
0929 02:09:19.257227 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:19.257252 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:19.257348 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:19.260189 (+ 2841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:19.260194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:19.260223 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:19.260799 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.260811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.262682 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.262688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.264206 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.264218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.266306 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.266315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.266493 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.266517 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:09:19.266620 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 02:09:19.266743 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":4,"thread_start_us":271,"threads_started":1}
I20260929 02:09:19.267731 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.257102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.11.182.195:39119)
0929 02:09:19.259645 (+ 2543us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.259653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.259683 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:09:19.259767 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.259774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.259992 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.260195 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:19.260980 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.262513 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.264358 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.265770 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.265970 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.266135 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.266681 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.266833 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2387,"thread_start_us":2251,"threads_started":1}
I20260929 02:09:19.268054 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.257283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:44835 (local address 127.0.0.1:36534)
0929 02:09:19.257973 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:19.257995 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:19.258102 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:19.260479 (+ 2377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:19.260484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:19.260513 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:19.261211 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.261228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.263059 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.263064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.265094 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.265109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.267241 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.267247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.267417 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.267450 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:09:19.267537 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 02:09:19.267684 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":142,"thread_start_us":671,"threads_started":1}
I20260929 02:09:19.268707 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.257813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.11.182.194:44835)
0929 02:09:19.259974 (+ 2161us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.259980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.260000 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:09:19.260069 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.260073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.260294 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.260493 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:19.261413 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.262929 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.265279 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.266726 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.266894 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.267057 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.268269 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.268432 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2014,"thread_start_us":1919,"threads_started":1}
I20260929 02:09:19.270152 26533 tablet_service.cc:1467] Processing CreateTablet for tablet 5efaa03b224b48759caa29396274b8e3 (DEFAULT_TABLE table=test-table [id=62c0f0e1f522443e86a12686d032b2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:19.271420 26533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efaa03b224b48759caa29396274b8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:19.271116 26796 tablet_service.cc:1467] Processing CreateTablet for tablet 5efaa03b224b48759caa29396274b8e3 (DEFAULT_TABLE table=test-table [id=62c0f0e1f522443e86a12686d032b2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:19.272048 26665 tablet_service.cc:1467] Processing CreateTablet for tablet 5efaa03b224b48759caa29396274b8e3 (DEFAULT_TABLE table=test-table [id=62c0f0e1f522443e86a12686d032b2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:19.272615 26796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efaa03b224b48759caa29396274b8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:19.273310 26665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efaa03b224b48759caa29396274b8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:19.293114 26877 tablet_bootstrap.cc:492] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: Bootstrap starting.
I20260929 02:09:19.293783 26878 tablet_bootstrap.cc:492] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: Bootstrap starting.
I20260929 02:09:19.293980 26879 tablet_bootstrap.cc:492] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: Bootstrap starting.
I20260929 02:09:19.298324 26879 tablet_bootstrap.cc:654] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:19.298322 26878 tablet_bootstrap.cc:654] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:19.299654 26878 log.cc:824] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:19.299638 26879 log.cc:824] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:19.302066 26877 tablet_bootstrap.cc:654] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:19.302717 26879 tablet_bootstrap.cc:492] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: No bootstrap required, opened a new log
I20260929 02:09:19.303007 26879 ts_tablet_manager.cc:1397] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:09:19.303329 26877 log.cc:824] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:19.304082 26878 tablet_bootstrap.cc:492] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: No bootstrap required, opened a new log
I20260929 02:09:19.304394 26878 ts_tablet_manager.cc:1397] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 02:09:19.306315 26877 tablet_bootstrap.cc:492] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: No bootstrap required, opened a new log
I20260929 02:09:19.306591 26877 ts_tablet_manager.cc:1397] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260929 02:09:19.314669 26879 raft_consensus.cc:348] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:19.315541 26879 raft_consensus.cc:374] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:19.315757 26879 raft_consensus.cc:729] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b79a13ce164a07b26d31a47215da6a, State: Initialized, Role: FOLLOWER
I20260929 02:09:19.315961 26877 raft_consensus.cc:348] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:19.316496 26877 raft_consensus.cc:374] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:19.316661 26877 raft_consensus.cc:729] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a958a7c37c4f3d94a22774d738219b, State: Initialized, Role: FOLLOWER
I20260929 02:09:19.316646 26879 consensus_queue.cc:260] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:19.316977 26878 raft_consensus.cc:348] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:19.317451 26878 raft_consensus.cc:374] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:19.317529 26598 heartbeater.cc:503] Master 127.11.182.254:41343 was elected leader, sending a full tablet report...
I20260929 02:09:19.317562 26878 raft_consensus.cc:729] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18fbd776335d4a3193af4ae08c70973e, State: Initialized, Role: FOLLOWER
I20260929 02:09:19.317468 26877 consensus_queue.cc:260] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:19.320230 26878 consensus_queue.cc:260] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:19.324689 26879 ts_tablet_manager.cc:1428] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: Time spent starting tablet: real 0.022s user 0.009s sys 0.010s
I20260929 02:09:19.324826 26877 ts_tablet_manager.cc:1428] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260929 02:09:19.325394 26730 heartbeater.cc:503] Master 127.11.182.254:41343 was elected leader, sending a full tablet report...
I20260929 02:09:19.325959 26861 heartbeater.cc:503] Master 127.11.182.254:41343 was elected leader, sending a full tablet report...
I20260929 02:09:19.326310 26878 ts_tablet_manager.cc:1428] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: Time spent starting tablet: real 0.022s user 0.017s sys 0.000s
I20260929 02:09:19.389932 26883 raft_consensus.cc:482] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:09:19.390213 26883 raft_consensus.cc:504] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:19.392052 26883 leader_election.cc:288] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08a958a7c37c4f3d94a22774d738219b (127.11.182.193:34845), 18fbd776335d4a3193af4ae08c70973e (127.11.182.195:39119)
I20260929 02:09:19.398380 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.392447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:59275 (local address 127.11.182.193:34845)
0929 02:09:19.392648 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.392653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.393255 (+ 602us) server_negotiation.cc:407] Connection header received
0929 02:09:19.393463 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.393469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.393553 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.393686 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:19.394238 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.394990 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.396681 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.397423 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.397597 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.397685 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.397947 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.398072 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:09:19.398470 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.392427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:34845 (local address 127.11.182.194:59275)
0929 02:09:19.393142 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:19.393167 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:19.393305 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:19.393711 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:19.393716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:19.393743 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:19.394088 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.394097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.395119 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.395134 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.396522 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.396533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.397631 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.397637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.397793 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.397816 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:09:19.397921 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:09:19.398044 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":387,"threads_started":1}
I20260929 02:09:19.398991 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.393249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:39119 (local address 127.11.182.194:60827)
0929 02:09:19.393870 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:19.393883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:19.394032 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:19.394362 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:19.394366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:19.394388 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:19.394705 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.394713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.395711 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.395716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.397052 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:19.397063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.398198 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.398205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:19.398364 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.398386 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:09:19.398498 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:09:19.398639 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":427,"threads_started":1}
I20260929 02:09:19.399221 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.393431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:60827 (local address 127.11.182.195:39119)
0929 02:09:19.393655 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.393661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.393966 (+ 305us) server_negotiation.cc:407] Connection header received
0929 02:09:19.394177 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.394181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.394235 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.394348 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:19.394884 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.395566 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.397200 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.397970 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.398151 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.398226 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.398845 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.398993 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:09:19.399565 26553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efaa03b224b48759caa29396274b8e3" candidate_uuid: "13b79a13ce164a07b26d31a47215da6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a958a7c37c4f3d94a22774d738219b" is_pre_election: true
I20260929 02:09:19.400116 26553 raft_consensus.cc:2454] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b79a13ce164a07b26d31a47215da6a in term 0.
I20260929 02:09:19.400767 26618 leader_election.cc:302] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a958a7c37c4f3d94a22774d738219b, 13b79a13ce164a07b26d31a47215da6a; no voters:
I20260929 02:09:19.400755 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efaa03b224b48759caa29396274b8e3" candidate_uuid: "13b79a13ce164a07b26d31a47215da6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18fbd776335d4a3193af4ae08c70973e" is_pre_election: true
I20260929 02:09:19.401182 26883 raft_consensus.cc:2781] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:09:19.401332 26883 raft_consensus.cc:482] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:09:19.401300 26816 raft_consensus.cc:2454] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b79a13ce164a07b26d31a47215da6a in term 0.
I20260929 02:09:19.401438 26883 raft_consensus.cc:3037] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:19.403810 26883 raft_consensus.cc:504] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:19.404371 26883 leader_election.cc:288] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [CANDIDATE]: Term 1 election: Requested vote from peers 08a958a7c37c4f3d94a22774d738219b (127.11.182.193:34845), 18fbd776335d4a3193af4ae08c70973e (127.11.182.195:39119)
I20260929 02:09:19.404755 26553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efaa03b224b48759caa29396274b8e3" candidate_uuid: "13b79a13ce164a07b26d31a47215da6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a958a7c37c4f3d94a22774d738219b"
I20260929 02:09:19.404806 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efaa03b224b48759caa29396274b8e3" candidate_uuid: "13b79a13ce164a07b26d31a47215da6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18fbd776335d4a3193af4ae08c70973e"
I20260929 02:09:19.404955 26553 raft_consensus.cc:3037] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:19.404994 26816 raft_consensus.cc:3037] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:19.407159 26553 raft_consensus.cc:2454] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b79a13ce164a07b26d31a47215da6a in term 1.
I20260929 02:09:19.407254 26816 raft_consensus.cc:2454] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b79a13ce164a07b26d31a47215da6a in term 1.
I20260929 02:09:19.407600 26618 leader_election.cc:302] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a958a7c37c4f3d94a22774d738219b, 13b79a13ce164a07b26d31a47215da6a; no voters:
I20260929 02:09:19.407863 26883 raft_consensus.cc:2781] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:09:19.408277 26883 raft_consensus.cc:686] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 1 LEADER]: Becoming Leader. State: Replica: 13b79a13ce164a07b26d31a47215da6a, State: Running, Role: LEADER
I20260929 02:09:19.408727 26883 consensus_queue.cc:237] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } }
I20260929 02:09:19.413399 26412 catalog_manager.cc:5495] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 13b79a13ce164a07b26d31a47215da6a (127.11.182.194). New cstate: current_term: 1 leader_uuid: "13b79a13ce164a07b26d31a47215da6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13b79a13ce164a07b26d31a47215da6a" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 44835 } health_report { overall_health: HEALTHY } } }
I20260929 02:09:19.434252 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.429333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.11.182.194:44835)
0929 02:09:19.429518 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.429522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.429537 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:09:19.429616 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.429619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.429697 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.429819 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:09:19.430107 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.430813 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.432250 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.432650 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.432737 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.433158 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:09:19.433701 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:09:19.433805 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.433878 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.433941 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 02:09:19.440563 26862 tablet.cc:2369] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:09:19.442793 26816 raft_consensus.cc:1261] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [term 1 FOLLOWER]: Refusing update from remote peer 13b79a13ce164a07b26d31a47215da6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:19.442956 26553 raft_consensus.cc:1261] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [term 1 FOLLOWER]: Refusing update from remote peer 13b79a13ce164a07b26d31a47215da6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:19.443794 26883 consensus_queue.cc:1035] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18fbd776335d4a3193af4ae08c70973e" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 39119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:19.444096 26888 consensus_queue.cc:1035] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a958a7c37c4f3d94a22774d738219b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 34845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:19.455663 26891 mvcc.cc:205] Tried to move back new op lower bound from 7334493222667563008 to 7334493222542172160. Current Snapshot: MvccSnapshot[applied={T|T < 7334493222667563008}]
I20260929 02:09:19.469678 26412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:09:19.469939 26412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:09:19.473412 26412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Sending DeleteTablet for 3 replicas of tablet 5efaa03b224b48759caa29396274b8e3
I20260929 02:09:19.474550 26533 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efaa03b224b48759caa29396274b8e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:19 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:60578
I20260929 02:09:19.474756 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 26450
I20260929 02:09:19.475363 26796 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efaa03b224b48759caa29396274b8e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:19 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:54792
I20260929 02:09:19.475881 26901 tablet_replica.cc:330] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: stopping tablet replica
I20260929 02:09:19.476107 26665 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efaa03b224b48759caa29396274b8e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:19 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:36534
I20260929 02:09:19.476257 26902 tablet_replica.cc:330] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: stopping tablet replica
I20260929 02:09:19.476336 26901 raft_consensus.cc:2229] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:09:19.476644 26902 raft_consensus.cc:2229] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:09:19.476814 26901 raft_consensus.cc:2258] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:19.477173 26902 raft_consensus.cc:2258] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:19.479290 26901 ts_tablet_manager.cc:1905] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:19.483551 26902 ts_tablet_manager.cc:1905] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:19.486397 26901 ts_tablet_manager.cc:1918] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:19.486559 26901 log.cc:1196] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/5efaa03b224b48759caa29396274b8e3
I20260929 02:09:19.487159 26901 ts_tablet_manager.cc:1939] T 5efaa03b224b48759caa29396274b8e3 P 08a958a7c37c4f3d94a22774d738219b: Deleting consensus metadata
I20260929 02:09:19.488937 26397 catalog_manager.cc:4841] TS 08a958a7c37c4f3d94a22774d738219b (127.11.182.193:34845): tablet 5efaa03b224b48759caa29396274b8e3 (table test-table [id=62c0f0e1f522443e86a12686d032b2fb]) successfully deleted
I20260929 02:09:19.489806 26903 tablet_replica.cc:330] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: stopping tablet replica
I20260929 02:09:19.490214 26903 raft_consensus.cc:2229] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:09:19.490918 26903 raft_consensus.cc:2258] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:19.493819 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.475661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.11.182.193:34845)
0929 02:09:19.476244 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.476251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.476268 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:09:19.476330 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.476335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.476418 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.476540 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:19.478503 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.480009 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.481863 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.482482 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.482613 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.485897 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:19.485912 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:19.485929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:19.485970 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:19.490676 (+ 4706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.491562 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.491571 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.491577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.491743 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.492131 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.492136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.492139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.492588 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:19.492884 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.493183 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.493429 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458}
I20260929 02:09:19.493815 26902 ts_tablet_manager.cc:1918] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:19.493968 26902 log.cc:1196] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/5efaa03b224b48759caa29396274b8e3
I20260929 02:09:19.494460 26902 ts_tablet_manager.cc:1939] T 5efaa03b224b48759caa29396274b8e3 P 18fbd776335d4a3193af4ae08c70973e: Deleting consensus metadata
I20260929 02:09:19.494165 26903 ts_tablet_manager.cc:1905] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:19.494774 26593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:19.499962 26397 catalog_manager.cc:4841] TS 18fbd776335d4a3193af4ae08c70973e (127.11.182.195:39119): tablet 5efaa03b224b48759caa29396274b8e3 (table test-table [id=62c0f0e1f522443e86a12686d032b2fb]) successfully deleted
I20260929 02:09:19.502292 26903 ts_tablet_manager.cc:1918] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:19.502468 26903 log.cc:1196] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/5efaa03b224b48759caa29396274b8e3
I20260929 02:09:19.503067 26903 ts_tablet_manager.cc:1939] T 5efaa03b224b48759caa29396274b8e3 P 13b79a13ce164a07b26d31a47215da6a: Deleting consensus metadata
I20260929 02:09:19.504626 26397 catalog_manager.cc:4841] TS 13b79a13ce164a07b26d31a47215da6a (127.11.182.194:44835): tablet 5efaa03b224b48759caa29396274b8e3 (table test-table [id=62c0f0e1f522443e86a12686d032b2fb]) successfully deleted
I20260929 02:09:19.584714 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 26450
I20260929 02:09:19.604226 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 26602
I20260929 02:09:19.619972 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.604958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.11.182.194:44835)
0929 02:09:19.605186 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.605194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.605257 (+ 63us) server_negotiation.cc:407] Connection header received
0929 02:09:19.605453 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.605457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.605521 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.605612 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:19.607352 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.608131 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.610044 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.610440 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.610532 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.613837 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:19.613851 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:19.613868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:19.613907 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:19.616942 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.617794 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.617800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.617802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.617966 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.618401 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.618404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.618406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.618871 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:19.619157 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.619467 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.619626 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:09:19.620668 26725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:19.732259 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 26602
I20260929 02:09:19.751708 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 26733
I20260929 02:09:19.765220 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:19.752491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.11.182.195:39119)
0929 02:09:19.752689 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:19.752697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:19.752718 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:09:19.752924 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:19.752933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:19.753002 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:19.753109 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:19.754408 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.755203 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.756476 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:19.756867 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:19.757029 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:19.759636 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:19.759648 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:19.759663 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:19.759699 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:19.762331 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.763139 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.763148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.763151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.763306 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:19.763674 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:19.763681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:19.763684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:19.764038 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:19.764331 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:19.764672 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 02:09:19.764894 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:09:19.765897 26856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:19.854962 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 26733
I20260929 02:09:19.871451 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 26380
I20260929 02:09:19.872613 26443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:19.984092 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 26380
I20260929 02:09:20.005640 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41343
--webserver_interface=127.11.182.254
--webserver_port=40947
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:20.329972 26908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:20.330271 26908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:20.330358 26908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:20.337450 26908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:09:20.337544 26908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:20.337590 26908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:20.337668 26908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:09:20.337731 26908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:09:20.347082 26908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:41343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=40947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:20.348301 26908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:20.349532 26908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:20.360925 26914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:20.360895 26916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:20.360911 26913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:20.361492 26908 server_base.cc:1034] running on GCE node
I20260929 02:09:20.362231 26908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:20.363570 26908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:20.364786 26908 hybrid_clock.cc:648] HybridClock initialized: now 1790647760364742 us; error 53 us; skew 500 ppm
I20260929 02:09:20.365097 26908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:20 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647760, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:20.368165 26908 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:09:20.369480 26908 webserver.cc:460] Webserver started at http://127.11.182.254:40947/ using document root <none> and password file <none>
I20260929 02:09:20.370105 26908 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:20.370196 26908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:20.375247 26908 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:09:20.377871 26923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:20.379184 26908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:09:20.379338 26908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "1bd51afd26cf462e838cce7450d5db04"
format_stamp: "Formatted at 2026-09-29 02:09:14 on dist-test-slave-q06p"
server_key: "cdedce427b15e9541b4520456e88e738"
server_key_iv: "f7cdf1e07b03848118c240a1cf3105ec"
server_key_version: "encryptionkey@0"
I20260929 02:09:20.380259 26908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:20.397047 26908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:20.401114 26908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:20.401430 26908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:20.419955 26908 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:41343
I20260929 02:09:20.419993 26974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:41343 every 8 connection(s)
I20260929 02:09:20.421365 26908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:09:20.425570 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 26908
I20260929 02:09:20.426317 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:34845
--local_ip_for_outbound_sockets=127.11.182.193
--tserver_master_addrs=127.11.182.254:41343
--webserver_port=41399
--webserver_interface=127.11.182.193
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:09:20.428308 26975 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:09:20.431604 26975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Bootstrap starting.
I20260929 02:09:20.461795 26975 log.cc:824] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:20.483866 26975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:09:20.484596 26975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04: Bootstrap complete.
I20260929 02:09:20.499308 26975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:20.500554 26975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd51afd26cf462e838cce7450d5db04, State: Initialized, Role: FOLLOWER
I20260929 02:09:20.501235 26975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:20.501518 26975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:09:20.501631 26975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:09:20.501804 26975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:09:20.507035 26975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:20.507597 26975 leader_election.cc:302] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bd51afd26cf462e838cce7450d5db04; no voters:
I20260929 02:09:20.508240 26975 leader_election.cc:288] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:09:20.508380 26979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:09:20.509662 26979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [term 3 LEADER]: Becoming Leader. State: Replica: 1bd51afd26cf462e838cce7450d5db04, State: Running, Role: LEADER
I20260929 02:09:20.510164 26975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:09:20.510183 26979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } }
I20260929 02:09:20.512450 26981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd51afd26cf462e838cce7450d5db04. Latest consensus state: current_term: 3 leader_uuid: "1bd51afd26cf462e838cce7450d5db04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } } }
I20260929 02:09:20.512811 26981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:20.513382 26980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1bd51afd26cf462e838cce7450d5db04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51afd26cf462e838cce7450d5db04" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 41343 } } }
I20260929 02:09:20.513644 26980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51afd26cf462e838cce7450d5db04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:20.513682 26985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:09:20.523703 26985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:09:20.525182 26985 catalog_manager.cc:1259] Loaded cluster ID: 5d7999b76f244520a8fa261034f798be
I20260929 02:09:20.525303 26985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 02:09:20.527336 26985 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f52a8ddb9ad google::LogMessage::Fail() at ??:0
@ 0x7f52a8dde88a google::LogMessage::SendToLog() at ??:0
@ 0x7f52a8ddb4de google::LogMessage::Flush() at ??:0
@ 0x7f52a8ddca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f52be80e512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f52be80b976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f52be893610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f52be893439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f52aba27345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f52aba2a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f52aba2a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f52aba05eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f52aecdf6db start_thread at ??:0
@ 0x7f52a7c7561f clone at ??:0
W20260929 02:09:20.836971 26977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:20.837275 26977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:20.837376 26977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:20.844578 26977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:20.844681 26977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:20.844789 26977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:09:20.854911 26977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:34845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=41399
--tserver_master_addrs=127.11.182.254:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:20.856082 26977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:20.857617 26977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:20.870894 27015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:20.870883 27017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:20.870785 27014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:20.876281 26977 server_base.cc:1034] running on GCE node
I20260929 02:09:20.876973 26977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:20.877950 26977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:20.879189 26977 hybrid_clock.cc:648] HybridClock initialized: now 1790647760879127 us; error 73 us; skew 500 ppm
I20260929 02:09:20.879559 26977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:20 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647760, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:20.882789 26977 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:09:20.884403 26977 webserver.cc:460] Webserver started at http://127.11.182.193:41399/ using document root <none> and password file <none>
I20260929 02:09:20.885142 26977 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:20.885298 26977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:20.890549 26977 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:09:20.895224 27024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:20.896596 26977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:09:20.896793 26977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "08a958a7c37c4f3d94a22774d738219b"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "38a82988837af5a85c38bdbbcf864763"
server_key_iv: "5fa6a455fe9613cb77e277a71ebf5cd0"
server_key_version: "encryptionkey@0"
I20260929 02:09:20.897783 26977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:20.913950 26977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:20.917711 26977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:20.917994 26977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:20.919045 26977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:20.920379 26977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:20.920542 26977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:20.920666 26977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:20.920756 26977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:20.954361 26977 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:34845
I20260929 02:09:20.954411 27136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:34845 every 8 connection(s)
I20260929 02:09:20.955920 26977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:09:20.960569 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:20.958771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:58661)
0929 02:09:20.959552 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:20.959757 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":296,"threads_started":1}
W20260929 02:09:20.961205 27137 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:41343 (0 consecutive failures): Network error: Failed to ping master at 127.11.182.254:41343: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
I20260929 02:09:20.961948 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:20.961664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:47805)
0929 02:09:20.961792 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:20.961852 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 02:09:20.962770 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:20.962484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:51809)
0929 02:09:20.962605 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:20.962676 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:09:20.962971 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 26977
W20260929 02:09:20.963007 27137 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:09:20.963588 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:44835
--local_ip_for_outbound_sockets=127.11.182.194
--tserver_master_addrs=127.11.182.254:41343
--webserver_port=45795
--webserver_interface=127.11.182.194
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:21.283927 27140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:21.284237 27140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:21.284416 27140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:21.291533 27140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:21.291692 27140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:21.291882 27140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:09:21.301295 27140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:44835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=45795
--tserver_master_addrs=127.11.182.254:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:21.302544 27140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:21.303851 27140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:21.317042 27145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:21.317008 27146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:21.317008 27148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:21.317729 27140 server_base.cc:1034] running on GCE node
I20260929 02:09:21.318346 27140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:21.319379 27140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:21.320600 27140 hybrid_clock.cc:648] HybridClock initialized: now 1790647761320547 us; error 61 us; skew 500 ppm
I20260929 02:09:21.320905 27140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:21 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647761, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:21.323971 27140 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:09:21.325335 27140 webserver.cc:460] Webserver started at http://127.11.182.194:45795/ using document root <none> and password file <none>
I20260929 02:09:21.325984 27140 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:21.326129 27140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:21.331825 27140 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:09:21.334380 27155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:21.335768 27140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:09:21.335947 27140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "13b79a13ce164a07b26d31a47215da6a"
format_stamp: "Formatted at 2026-09-29 02:09:15 on dist-test-slave-q06p"
server_key: "94aa4f71381d21555b943134640c38fe"
server_key_iv: "62feae85874016e2d839c9438fbf9c29"
server_key_version: "encryptionkey@0"
I20260929 02:09:21.336911 27140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:21.357750 27140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:21.361563 27140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:21.361850 27140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:21.362823 27140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:21.364208 27140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:21.364328 27140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:21.364467 27140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:21.364559 27140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:21.397746 27140 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:44835
I20260929 02:09:21.397830 27267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:44835 every 8 connection(s)
I20260929 02:09:21.399286 27140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:09:21.404034 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:21.402304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:34285)
0929 02:09:21.403056 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:21.403270 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":297,"threads_started":1}
W20260929 02:09:21.404589 27268 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:41343 (0 consecutive failures): Network error: Failed to ping master at 127.11.182.254:41343: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
I20260929 02:09:21.404995 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 27140
I20260929 02:09:21.405563 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:21.405089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:43259)
0929 02:09:21.405312 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:21.405395 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260929 02:09:21.405786 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:39119
--local_ip_for_outbound_sockets=127.11.182.195
--tserver_master_addrs=127.11.182.254:41343
--webserver_port=39725
--webserver_interface=127.11.182.195
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:09:21.406581 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:21.406145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:53545)
0929 02:09:21.406317 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:21.406417 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:09:21.407001 27268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:09:21.731915 27271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:21.732225 27271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:21.732395 27271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:21.739631 27271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:21.739782 27271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:21.739984 27271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:09:21.749555 27271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:39119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=39725
--tserver_master_addrs=127.11.182.254:41343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:21.750725 27271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:21.752018 27271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:21.764882 27276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:21.764820 27279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:21.764847 27277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:21.765764 27271 server_base.cc:1034] running on GCE node
I20260929 02:09:21.766338 27271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:21.767256 27271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:21.768497 27271 hybrid_clock.cc:648] HybridClock initialized: now 1790647761768436 us; error 47 us; skew 500 ppm
I20260929 02:09:21.768800 27271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:21 dist-test-slave-q06p krb5kdc[25831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647761, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:21.771804 27271 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:09:21.773214 27271 webserver.cc:460] Webserver started at http://127.11.182.195:39725/ using document root <none> and password file <none>
I20260929 02:09:21.773888 27271 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:21.774055 27271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:21.779886 27271 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:09:21.782433 27286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:21.783741 27271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:09:21.783914 27271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "18fbd776335d4a3193af4ae08c70973e"
format_stamp: "Formatted at 2026-09-29 02:09:16 on dist-test-slave-q06p"
server_key: "576ffed1c4e15ae331860bd498a81b72"
server_key_iv: "44268dc02bebb4e069719d6b23757694"
server_key_version: "encryptionkey@0"
I20260929 02:09:21.784848 27271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:21.800184 27271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:21.804342 27271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:21.804620 27271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:21.805517 27271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:21.806721 27271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:21.806839 27271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:21.807010 27271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:21.807109 27271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:21.840152 27398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:39119 every 8 connection(s)
I20260929 02:09:21.840252 27271 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:39119
I20260929 02:09:21.841881 27271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:09:21.846706 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:21.844878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:53043)
0929 02:09:21.845702 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:21.845901 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":313,"threads_started":1}
W20260929 02:09:21.847393 27399 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:41343 (0 consecutive failures): Network error: Failed to ping master at 127.11.182.254:41343: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
I20260929 02:09:21.847613 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 27271
I20260929 02:09:21.848189 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:21.847893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:55193)
0929 02:09:21.848034 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:21.848093 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 02:09:21.848964 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:21.848670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:59965)
0929 02:09:21.848796 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:21.848870 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:09:21.849238 27399 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:09:21.964262 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:21.963447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:55571)
0929 02:09:21.963889 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:21.964028 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":211,"threads_started":1}
I20260929 02:09:22.408222 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:22.407422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:42921)
0929 02:09:22.407906 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:22.408014 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
I20260929 02:09:22.850507 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:22.849681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:49477)
0929 02:09:22.850149 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:22.850275 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
I20260929 02:09:22.965684 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:22.965036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:46085)
0929 02:09:22.965441 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:22.965517 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":201,"threads_started":1}
I20260929 02:09:23.409683 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:23.409055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:45123)
0929 02:09:23.409463 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:23.409532 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":208,"threads_started":1}
I20260929 02:09:23.852074 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:23.851363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:40993)
0929 02:09:23.851805 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:23.851884 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260929 02:09:23.967245 27409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:23.966570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:41543)
0929 02:09:23.967006 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:23.967081 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":207,"threads_started":1}
I20260929 02:09:24.411260 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:24.410572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:47751)
0929 02:09:24.411012 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:24.411089 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":207,"threads_started":1}
I20260929 02:09:24.853595 27411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:24.852882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:49535)
0929 02:09:24.853328 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:24.853406 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":257,"threads_started":1}
I20260929 02:09:24.968736 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:24.968076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:44561)
0929 02:09:24.968473 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:24.968553 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":200,"threads_started":1}
I20260929 02:09:25.412899 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:25.412073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:60175)
0929 02:09:25.412574 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:25.412658 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":238,"threads_started":1}
I20260929 02:09:25.855333 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:25.854503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:39817)
0929 02:09:25.855058 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:25.855135 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":294,"threads_started":1}
I20260929 02:09:25.970261 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:25.969606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:51745)
0929 02:09:25.970025 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:25.970100 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260929 02:09:26.414572 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:26.413830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:41261)
0929 02:09:26.414308 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:26.414388 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260929 02:09:26.857122 27417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:26.856274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:44355)
0929 02:09:26.856849 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:26.856927 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":305,"threads_started":1}
I20260929 02:09:26.971846 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:26.971187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:36467)
0929 02:09:26.971596 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:26.971683 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":204,"threads_started":1}
I20260929 02:09:27.416158 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:27.415472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:57477)
0929 02:09:27.415898 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:27.416001 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":219,"threads_started":1}
I20260929 02:09:27.858647 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:27.857949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:35989)
0929 02:09:27.858381 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:27.858466 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260929 02:09:27.973520 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:27.972692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:34243)
0929 02:09:27.973200 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:27.973290 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":215,"threads_started":1}
I20260929 02:09:28.418009 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:28.417145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:58363)
0929 02:09:28.417687 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:28.417786 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":241,"threads_started":1}
I20260929 02:09:28.860622 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:28.859777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:43547)
0929 02:09:28.860305 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:28.860391 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":247,"threads_started":1}
I20260929 02:09:28.975559 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:28.974598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:32915)
0929 02:09:28.975188 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:28.975331 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":302,"threads_started":1}
I20260929 02:09:29.419708 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:29.419019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:41239)
0929 02:09:29.419468 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:29.419551 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":236,"threads_started":1}
I20260929 02:09:29.862491 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:29.861709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:55841)
0929 02:09:29.862244 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:29.862327 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":279,"threads_started":1}
I20260929 02:09:29.977195 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:29.976489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:53637)
0929 02:09:29.976933 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:29.977014 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":215,"threads_started":1}
I20260929 02:09:30.421592 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:30.420686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:38401)
0929 02:09:30.421309 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:30.421395 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":331,"threads_started":1}
I20260929 02:09:30.864177 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:30.863435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:34965)
0929 02:09:30.863913 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:30.864004 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260929 02:09:30.979015 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:30.978164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:56331)
0929 02:09:30.978660 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:30.978749 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260929 02:09:31.423494 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:31.422618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:49361)
0929 02:09:31.423220 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:31.423311 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":360,"threads_started":1}
I20260929 02:09:31.865808 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:31.865106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:57777)
0929 02:09:31.865544 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:31.865641 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":249,"threads_started":1}
I20260929 02:09:31.980480 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:31.979829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:42981)
0929 02:09:31.980228 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:31.980313 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":204,"threads_started":1}
I20260929 02:09:32.425571 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:32.424452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:40621)
0929 02:09:32.425201 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:32.425312 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":528,"threads_started":1}
I20260929 02:09:32.867547 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:32.866721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:37527)
0929 02:09:32.867291 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:32.867377 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":390,"threads_started":1}
I20260929 02:09:32.982041 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:32.981403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:54071)
0929 02:09:32.981810 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:32.981891 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":211,"threads_started":1}
I20260929 02:09:33.427593 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:33.426804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:41455)
0929 02:09:33.427313 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:33.427427 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260929 02:09:33.869481 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:33.868733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:52297)
0929 02:09:33.869247 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:33.869332 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":289,"threads_started":1}
I20260929 02:09:33.983928 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:33.983124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:60077)
0929 02:09:33.983646 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:33.983740 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":249,"threads_started":1}
I20260929 02:09:34.429374 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:34.428583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:56861)
0929 02:09:34.429094 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:34.429190 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":257,"threads_started":1}
I20260929 02:09:34.871381 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:34.870583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:53957)
0929 02:09:34.871113 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:34.871206 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":299,"threads_started":1}
I20260929 02:09:34.985622 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:34.984890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:48693)
0929 02:09:34.985326 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:34.985438 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":213,"threads_started":1}
I20260929 02:09:35.430943 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:35.430254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:39669)
0929 02:09:35.430662 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:35.430742 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":205,"threads_started":1}
I20260929 02:09:35.872922 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:35.872263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:50213)
0929 02:09:35.872670 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:35.872751 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":212,"threads_started":1}
I20260929 02:09:35.987418 27445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:35.986675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.193:53727)
0929 02:09:35.987145 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:35.987240 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":271,"threads_started":1}
I20260929 02:09:36.432741 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:36.432022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.194:60039)
0929 02:09:36.432459 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:36.432566 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":234,"threads_started":1}
I20260929 02:09:36.849946 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 26977
I20260929 02:09:36.868256 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:36.851021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.11.182.193:34845)
0929 02:09:36.851717 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:36.851738 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:36.851788 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:09:36.853920 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:36.853929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:36.854353 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:36.854694 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:36.856173 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:36.857384 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:36.858642 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:36.859373 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:36.859562 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:36.862147 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:36.862155 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:36.862170 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:36.862206 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:36.865099 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:36.865749 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:36.865763 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:36.865770 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:36.865935 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:36.866309 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:36.866313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:36.866315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:36.866810 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:36.867181 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:36.867546 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 02:09:36.867958 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":342,"threads_started":1}
I20260929 02:09:36.869660 27132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:36.874905 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:36.874089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:41343 (local address 127.11.182.195:54695)
0929 02:09:36.874584 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:36.874691 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":237,"threads_started":1}
I20260929 02:09:36.954173 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 26977
I20260929 02:09:36.971017 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 27140
I20260929 02:09:36.987861 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:36.971841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.11.182.194:44835)
0929 02:09:36.972425 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:36.972437 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:36.972474 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:09:36.974005 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:36.974014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:36.974427 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:36.974714 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:36.976197 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:36.977389 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:36.978628 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:36.979329 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:36.979460 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:36.981945 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:36.981953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:36.981966 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:36.982002 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:36.984863 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:36.985486 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:36.985492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:36.985497 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:36.985662 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:36.986080 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:36.986083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:36.986085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:36.986540 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:36.986780 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:36.987266 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 02:09:36.987583 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":319,"threads_started":1}
I20260929 02:09:36.989501 27263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:37.071631 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 27140
I20260929 02:09:37.088838 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 27271
I20260929 02:09:37.106328 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:37.089761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.11.182.195:39119)
0929 02:09:37.090412 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:37.090427 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:37.090468 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:09:37.092141 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:37.092149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:37.092571 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:37.092905 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:37.094269 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:37.095594 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:37.096838 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:37.097480 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:37.097691 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:37.100329 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:37.100337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:37.100351 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:37.100388 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:37.103136 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:37.103794 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:37.103803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:37.103815 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:37.103980 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:37.104341 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:37.104348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:37.104351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:37.104859 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:37.105175 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:37.105635 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 02:09:37.106035 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":283,"threads_started":1}
I20260929 02:09:37.107779 27394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:37.192766 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 27271
2026-09-29T02:09:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24211 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:09:37 dist-test-slave-q06p krb5kdc[27456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:09:37 dist-test-slave-q06p krb5kdc[27456](info): set up 2 sockets
Sep 29 02:09:37 dist-test-slave-q06p krb5kdc[27456](info): commencing operation
krb5kdc: starting...
W20260929 02:09:38.291903 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:09:38 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:09:38.518779 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:43017
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:43017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:38.847734 27469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:38.847997 27469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:38.848171 27469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:38.855151 27469 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:09:38.855235 27469 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:09:38.855316 27469 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:09:38.855422 27469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:09:38.855507 27469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:38.855631 27469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:38.855700 27469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:09:38.855777 27469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:09:38.865096 27469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:43017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:38.866338 27469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:38.867667 27469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:38.878909 27474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:38.878971 27475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:38.879005 27477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:38.879837 27469 server_base.cc:1034] running on GCE node
I20260929 02:09:38.880605 27469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:38.882000 27469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:38.883234 27469 hybrid_clock.cc:648] HybridClock initialized: now 1790647778883169 us; error 48 us; skew 500 ppm
I20260929 02:09:38.883562 27469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:38 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647778, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:38.886513 27469 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:09:38.887921 27469 webserver.cc:460] Webserver started at http://127.11.182.254:38483/ using document root <none> and password file <none>
I20260929 02:09:38.888574 27469 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:38.888657 27469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:38.888973 27469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:38.891870 27469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "b541d1ba3284498ba15df4f3d1172d60"
format_stamp: "Formatted at 2026-09-29 02:09:38 on dist-test-slave-q06p"
server_key: "e1a771b9a219b0795d9872a0ac737c1b"
server_key_iv: "595ce0fc216013efcdea1052e3f12834"
server_key_version: "encryptionkey@0"
I20260929 02:09:38.892580 27469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "b541d1ba3284498ba15df4f3d1172d60"
format_stamp: "Formatted at 2026-09-29 02:09:38 on dist-test-slave-q06p"
server_key: "e1a771b9a219b0795d9872a0ac737c1b"
server_key_iv: "595ce0fc216013efcdea1052e3f12834"
server_key_version: "encryptionkey@0"
I20260929 02:09:38.897630 27469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 02:09:38.901014 27484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:38.902376 27469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:09:38.902542 27469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "b541d1ba3284498ba15df4f3d1172d60"
format_stamp: "Formatted at 2026-09-29 02:09:38 on dist-test-slave-q06p"
server_key: "e1a771b9a219b0795d9872a0ac737c1b"
server_key_iv: "595ce0fc216013efcdea1052e3f12834"
server_key_version: "encryptionkey@0"
I20260929 02:09:38.902799 27469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:38.919399 27469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:38.923192 27469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:38.923442 27469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:38.941133 27469 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:43017
I20260929 02:09:38.941167 27535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:43017 every 8 connection(s)
I20260929 02:09:38.942407 27469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:09:38.945948 27536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:38.950133 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 27469
I20260929 02:09:38.950349 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:09:38.951195 11995 external_mini_cluster.cc:1428] Setting key cb8d5b9388339a5377b2588a86595631
I20260929 02:09:38.957690 27536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Bootstrap starting.
I20260929 02:09:38.961261 27536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:38.962479 27536 log.cc:824] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:38.966081 27536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: No bootstrap required, opened a new log
Sep 29 02:09:38 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:38.973732 27536 raft_consensus.cc:348] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:09:38.974112 27536 raft_consensus.cc:374] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:38.974251 27536 raft_consensus.cc:729] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b541d1ba3284498ba15df4f3d1172d60, State: Initialized, Role: FOLLOWER
I20260929 02:09:38.974906 27536 consensus_queue.cc:260] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:09:38.975147 27536 raft_consensus.cc:388] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:09:38.975301 27536 raft_consensus.cc:482] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:09:38.975410 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:38.953197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.11.182.254:43017)
0929 02:09:38.953926 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:38.953938 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:38.953981 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:09:38.956140 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:38.956163 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:38.956527 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:38.956844 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:38.958675 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:38.960200 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:38.962256 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:38.963278 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:38.963506 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:38.968101 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:38.968110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:38.968127 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:38.968169 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:38.971348 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:38.971998 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:38.972004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:38.972013 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:38.972177 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:38.972556 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:38.972562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:38.972573 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:38.973088 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:38.973391 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:38.973873 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 02:09:38.974303 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":346,"threads_started":1}
I20260929 02:09:38.975482 27536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:38.977777 27536 raft_consensus.cc:504] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:09:38.978292 27536 leader_election.cc:302] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b541d1ba3284498ba15df4f3d1172d60; no voters:
I20260929 02:09:38.978906 27536 leader_election.cc:288] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:09:38.979010 27541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:09:38.979749 27541 raft_consensus.cc:686] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 1 LEADER]: Becoming Leader. State: Replica: b541d1ba3284498ba15df4f3d1172d60, State: Running, Role: LEADER
I20260929 02:09:38.980258 27541 consensus_queue.cc:237] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:09:38.980926 27536 sys_catalog.cc:564] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:09:38.982635 27543 sys_catalog.cc:455] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b541d1ba3284498ba15df4f3d1172d60. Latest consensus state: current_term: 1 leader_uuid: "b541d1ba3284498ba15df4f3d1172d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } } }
I20260929 02:09:38.983650 27543 sys_catalog.cc:458] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:38.983522 27542 sys_catalog.cc:455] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b541d1ba3284498ba15df4f3d1172d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } } }
I20260929 02:09:38.983994 27542 sys_catalog.cc:458] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: This master's current role is: LEADER
I20260929 02:09:38.991546 27547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:09:38.996984 27547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:09:39.004864 27547 catalog_manager.cc:1347] Generated new cluster ID: e5f5e719a6ef47e2adf8de866030118e
I20260929 02:09:39.004984 27547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:09:39.014647 27547 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:09:39.015915 27547 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:09:39.026142 27547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Generated new TSK 0
I20260929 02:09:39.027014 27547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:09:39.085692 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:43017
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:39.435413 27562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:39.435725 27562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:39.435892 27562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:39.442960 27562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:39.443135 27562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:39.443329 27562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:09:39.452728 27562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:39.453984 27562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:39.455282 27562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:39.468514 27568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:39.468655 27567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:39.468662 27570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:39.469637 27562 server_base.cc:1034] running on GCE node
I20260929 02:09:39.470170 27562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:39.471057 27562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:39.472275 27562 hybrid_clock.cc:648] HybridClock initialized: now 1790647779472229 us; error 53 us; skew 500 ppm
I20260929 02:09:39.472584 27562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:39 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647779, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:39.475561 27562 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:09:39.477174 27562 webserver.cc:460] Webserver started at http://127.11.182.193:40973/ using document root <none> and password file <none>
I20260929 02:09:39.477840 27562 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:39.477970 27562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:39.478255 27562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:39.481108 27562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "df5a301f17d8465b93d8b2c392525f04"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "b9923cf3ab0898074010c409ef3beeb7"
server_key_iv: "362fe4ee734e2204cc690d45e5695492"
server_key_version: "encryptionkey@0"
I20260929 02:09:39.481848 27562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "df5a301f17d8465b93d8b2c392525f04"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "b9923cf3ab0898074010c409ef3beeb7"
server_key_iv: "362fe4ee734e2204cc690d45e5695492"
server_key_version: "encryptionkey@0"
I20260929 02:09:39.486670 27562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 02:09:39.489840 27577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:39.491246 27562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:09:39.491400 27562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "df5a301f17d8465b93d8b2c392525f04"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "b9923cf3ab0898074010c409ef3beeb7"
server_key_iv: "362fe4ee734e2204cc690d45e5695492"
server_key_version: "encryptionkey@0"
I20260929 02:09:39.491588 27562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:39.509965 27562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:39.513594 27562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:39.513861 27562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:39.514730 27562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:39.516028 27562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:39.516124 27562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:39.516266 27562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:39.516332 27562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:39.548352 27562 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:40621
I20260929 02:09:39.548434 27689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:40621 every 8 connection(s)
I20260929 02:09:39.549599 27562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:09:39.550069 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 27562
I20260929 02:09:39.550318 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:09:39.551023 11995 external_mini_cluster.cc:1428] Setting key 93b816d98122b22d6a3aee23c511c49d
Sep 29 02:09:39 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647779, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:39.571439 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:39.552493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:49199 (local address 127.11.182.254:43017)
0929 02:09:39.553054 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:39.553059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:39.554014 (+ 955us) server_negotiation.cc:407] Connection header received
0929 02:09:39.555842 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:39.555847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:39.555960 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:39.556071 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:39.557517 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:39.558212 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:39.559832 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:39.560231 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:39.560360 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:39.564027 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:39.564042 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:39.564050 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:39.564095 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:39.567774 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:39.568565 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:39.568574 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:39.568581 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:39.568738 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:39.569258 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:39.569271 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:39.569278 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:39.569682 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:39.569845 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:39.570616 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 02:09:39.570912 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":234,"threads_started":1}
I20260929 02:09:39.572034 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:39.553167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:49199)
0929 02:09:39.553875 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:39.553919 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:39.555595 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:39.556293 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:39.556301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:39.556823 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:39.557345 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:39.557360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:39.558396 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:39.558400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:39.559684 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:39.559696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:39.560463 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:39.560470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:39.560687 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:39.561392 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:39.561419 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:39.563784 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:39.567950 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:39.567957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:39.567971 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:39.568424 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:39.568892 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:39.568896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:39.568899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:39.569086 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:39.569834 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:39.569840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:39.570208 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 02:09:39.570450 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 02:09:39.570922 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":302,"threads_started":1}
I20260929 02:09:39.573786 27690 heartbeater.cc:347] Connected to a master server at 127.11.182.254:43017
I20260929 02:09:39.574198 27690 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:39.575181 27690 heartbeater.cc:511] Master 127.11.182.254:43017 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:09:39.577569 27501 ts_manager.cc:194] Registered new tserver with Master: df5a301f17d8465b93d8b2c392525f04 (127.11.182.193:40621)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
I20260929 02:09:39.579273 27501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:49199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:09:39.606617 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:43017
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:39.924921 27696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:39.925215 27696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:39.925379 27696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:39.932410 27696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:39.932541 27696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:39.932710 27696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:09:39.942039 27696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:39.943243 27696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:39.944480 27696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:39.957332 27701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:39.957402 27704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:39.957428 27702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:39.960251 27696 server_base.cc:1034] running on GCE node
I20260929 02:09:39.960793 27696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:39.961587 27696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:39.962771 27696 hybrid_clock.cc:648] HybridClock initialized: now 1790647779962737 us; error 25 us; skew 500 ppm
I20260929 02:09:39.963060 27696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:39 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647779, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:39.965929 27696 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:09:39.967403 27696 webserver.cc:460] Webserver started at http://127.11.182.194:46813/ using document root <none> and password file <none>
I20260929 02:09:39.968035 27696 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:39.968159 27696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:39.968441 27696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:39.971153 27696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "987f2bc887284e84a9fd63bf4f9483bd"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "0d0c102ae954fd1898aa0cadd5c26839"
server_key_iv: "fd44a7b52a63271340640e222d2fbfad"
server_key_version: "encryptionkey@0"
I20260929 02:09:39.971854 27696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "987f2bc887284e84a9fd63bf4f9483bd"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "0d0c102ae954fd1898aa0cadd5c26839"
server_key_iv: "fd44a7b52a63271340640e222d2fbfad"
server_key_version: "encryptionkey@0"
I20260929 02:09:39.976553 27696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:09:39.979707 27711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:39.980868 27696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:09:39.981033 27696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "987f2bc887284e84a9fd63bf4f9483bd"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "0d0c102ae954fd1898aa0cadd5c26839"
server_key_iv: "fd44a7b52a63271340640e222d2fbfad"
server_key_version: "encryptionkey@0"
I20260929 02:09:39.981220 27696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:40.018481 27696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:40.022208 27696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:40.022493 27696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:40.023443 27696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:40.024768 27696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:40.024860 27696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:40.025002 27696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:40.025080 27696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:40.057919 27696 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:45041
I20260929 02:09:40.057998 27823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:45041 every 8 connection(s)
I20260929 02:09:40.059203 27696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:09:40.067955 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 27696
I20260929 02:09:40.068130 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:09:40.068573 11995 external_mini_cluster.cc:1428] Setting key 27263a00c37ed732b2802687ffe84213
Sep 29 02:09:40 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647779, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:40.079210 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.061595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:40547 (local address 127.11.182.254:43017)
0929 02:09:40.061768 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.061772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.062829 (+ 1057us) server_negotiation.cc:407] Connection header received
0929 02:09:40.064613 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.064618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.064706 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.064814 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:40.066235 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.067030 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.068643 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.069059 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.069200 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.073038 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:40.073043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:40.073048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:40.073079 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:40.075786 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.076670 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.076673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.076675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.076793 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.077245 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.077248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.077250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.077481 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:40.077592 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.078445 (+ 853us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.078741 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:09:40.079766 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.062051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:40547)
0929 02:09:40.062675 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:40.062713 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:40.064348 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:40.065050 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:40.065058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:40.065536 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:40.066074 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.066090 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.067174 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.067179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.068513 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.068524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.069279 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.069286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.069514 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.070171 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:40.070191 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:40.072848 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:40.075910 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:40.075921 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:40.075939 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:40.076566 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:40.076905 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:40.076912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:40.076918 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:40.077143 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:40.077599 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:40.077609 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:40.078053 (+ 444us) client_negotiation.cc:769] Sending connection context
0929 02:09:40.078332 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 02:09:40.078720 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":241,"threads_started":1}
I20260929 02:09:40.081158 27824 heartbeater.cc:347] Connected to a master server at 127.11.182.254:43017
I20260929 02:09:40.081436 27824 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:40.082067 27824 heartbeater.cc:511] Master 127.11.182.254:43017 requested a full tablet report, sending...
I20260929 02:09:40.083551 27501 ts_manager.cc:194] Registered new tserver with Master: 987f2bc887284e84a9fd63bf4f9483bd (127.11.182.194:45041)
I20260929 02:09:40.084391 27501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:40547
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:09:40.122820 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:43017
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:09:40.440008 27829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:09:40.440328 27829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:09:40.440507 27829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:09:40.447567 27829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:09:40.447707 27829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:09:40.447886 27829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:09:40.457201 27829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:09:40.458424 27829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:09:40.459712 27829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:09:40.472681 27834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:40.472636 27835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:09:40.472591 27837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:09:40.473433 27829 server_base.cc:1034] running on GCE node
I20260929 02:09:40.473951 27829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:09:40.474731 27829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:09:40.475916 27829 hybrid_clock.cc:648] HybridClock initialized: now 1790647780475878 us; error 58 us; skew 500 ppm
I20260929 02:09:40.476212 27829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:09:40 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647780, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:09:40.479152 27829 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:09:40.480508 27829 webserver.cc:460] Webserver started at http://127.11.182.195:41945/ using document root <none> and password file <none>
I20260929 02:09:40.481120 27829 fs_manager.cc:359] Metadata directory not provided
I20260929 02:09:40.481334 27829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:09:40.481618 27829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:09:40.484668 27829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "0a9def730ced4539a20930b721a7372b"
format_stamp: "Formatted at 2026-09-29 02:09:40 on dist-test-slave-q06p"
server_key: "865f09701c31acecba2d9a3444629e9f"
server_key_iv: "4f637059de7e5d3db1a6dda6ef9d6f2e"
server_key_version: "encryptionkey@0"
I20260929 02:09:40.485365 27829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "0a9def730ced4539a20930b721a7372b"
format_stamp: "Formatted at 2026-09-29 02:09:40 on dist-test-slave-q06p"
server_key: "865f09701c31acecba2d9a3444629e9f"
server_key_iv: "4f637059de7e5d3db1a6dda6ef9d6f2e"
server_key_version: "encryptionkey@0"
I20260929 02:09:40.490020 27829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:09:40.493213 27844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:40.494443 27829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:09:40.494611 27829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "0a9def730ced4539a20930b721a7372b"
format_stamp: "Formatted at 2026-09-29 02:09:40 on dist-test-slave-q06p"
server_key: "865f09701c31acecba2d9a3444629e9f"
server_key_iv: "4f637059de7e5d3db1a6dda6ef9d6f2e"
server_key_version: "encryptionkey@0"
I20260929 02:09:40.494803 27829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:09:40.512189 27829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:09:40.516343 27829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:09:40.516613 27829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:09:40.517503 27829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:09:40.519009 27829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:09:40.519112 27829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:40.519250 27829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:09:40.519318 27829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:09:40.550312 27829 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:44701
I20260929 02:09:40.550421 27956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:44701 every 8 connection(s)
I20260929 02:09:40.551616 27829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:09:40.554582 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 27829
I20260929 02:09:40.554762 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:09:40.555328 11995 external_mini_cluster.cc:1428] Setting key ac75235a361b86c69007b01e6e48b4b5
Sep 29 02:09:40 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647780, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:09:40.571256 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.554385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:53859 (local address 127.11.182.254:43017)
0929 02:09:40.554607 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.554614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.555883 (+ 1269us) server_negotiation.cc:407] Connection header received
0929 02:09:40.558142 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.558149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.558218 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.558289 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:40.559779 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.560506 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.561936 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.562355 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.562453 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.565647 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:40.565651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:40.565655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:40.565684 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:40.568092 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.568770 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.568775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.568778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.568948 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.569410 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.569414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.569416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.569734 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:40.569892 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.570646 (+ 754us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.570896 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:09:40.572196 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.554928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:53859)
0929 02:09:40.555731 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:40.555781 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:40.557867 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:40.558528 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:40.558535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:40.559098 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:09:40.559628 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.559644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.560636 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.560640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.561820 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.561830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.562532 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.562536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.562720 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.563344 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:09:40.563377 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:09:40.565460 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:09:40.568217 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:40.568223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:40.568234 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:40.568646 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:40.569073 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:09:40.569078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:09:40.569082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:09:40.569295 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:09:40.569857 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:09:40.569866 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:09:40.570244 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 02:09:40.570573 (+ 329us) client_negotiation.cc:240] Negotiation successful
0929 02:09:40.571107 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":291,"threads_started":1}
I20260929 02:09:40.573450 27957 heartbeater.cc:347] Connected to a master server at 127.11.182.254:43017
I20260929 02:09:40.573773 27957 heartbeater.cc:464] Registering TS with master...
I20260929 02:09:40.574465 27957 heartbeater.cc:511] Master 127.11.182.254:43017 requested a full tablet report, sending...
I20260929 02:09:40.575853 27501 ts_manager.cc:194] Registered new tserver with Master: 0a9def730ced4539a20930b721a7372b (127.11.182.195:44701)
I20260929 02:09:40.576596 27501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:53859
I20260929 02:09:40.582471 27690 heartbeater.cc:503] Master 127.11.182.254:43017 was elected leader, sending a full tablet report...
I20260929 02:09:40.582603 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:09:40.600201 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.587485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.11.182.254:43017)
0929 02:09:40.587642 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.587647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.588158 (+ 511us) server_negotiation.cc:407] Connection header received
0929 02:09:40.588357 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.588362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.588423 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.588519 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:40.589931 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.590692 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.592101 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.592515 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.592656 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.595061 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:40.595073 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:40.595076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:40.595107 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:40.597529 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.598165 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.598172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.598175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.598302 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.598703 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.598711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.598714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.599094 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:40.599349 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.599632 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.599843 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:09:40.604084 27501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:09:40.610765 27501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:09:40.650225 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.622099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:40621 (local address 127.0.0.1:59602)
0929 02:09:40.636992 (+ 14893us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:40.637055 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:40.637281 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:40.639516 (+ 2235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:40.639528 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:40.639572 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:40.640112 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.640133 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.644778 (+ 4645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.644787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.647767 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.647782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.649106 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.649112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.649292 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.649347 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 02:09:40.649492 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 02:09:40.649717 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14587,"mutex_wait_us":10,"thread_start_us":15408,"threads_started":1}
I20260929 02:09:40.650992 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.622425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.11.182.193:40621)
0929 02:09:40.638686 (+ 16261us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.638699 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.638737 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:09:40.638893 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.638905 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.639247 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.639495 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:40.640435 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.644500 (+ 4065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.647971 (+ 3471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.648902 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.649106 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.649279 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.650004 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.650220 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16041,"thread_start_us":15887,"threads_started":1}
I20260929 02:09:40.651098 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.622317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:44701 (local address 127.0.0.1:52124)
0929 02:09:40.640304 (+ 17987us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:40.640327 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:40.640569 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:40.641633 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:40.641640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:40.641674 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:40.642228 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.642242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.644763 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.644771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.647775 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.647783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.649649 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.649656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.649880 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.650445 (+ 565us) client_negotiation.cc:769] Sending connection context
0929 02:09:40.650533 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:09:40.650668 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17809,"mutex_wait_us":999,"thread_start_us":17621,"threads_started":1}
I20260929 02:09:40.651853 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.622582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.11.182.195:44701)
0929 02:09:40.637904 (+ 15322us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.637915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.640447 (+ 2532us) server_negotiation.cc:407] Connection header received
0929 02:09:40.640773 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.640784 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.641076 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.641258 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:40.642480 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.644542 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.647944 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.649415 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.649602 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.651258 (+ 1656us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.651375 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.651541 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15105,"thread_start_us":15329,"threads_started":1}
I20260929 02:09:40.654402 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.624363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:45041 (local address 127.0.0.1:45344)
0929 02:09:40.646131 (+ 21768us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:40.646155 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:40.646282 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:40.647029 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:40.647034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:40.647064 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:40.647601 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.647611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.649492 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.649497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.651759 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.651774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.653515 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.653521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.653713 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.653747 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:09:40.653859 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:09:40.653985 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20889,"mutex_wait_us":1,"thread_start_us":20651,"threads_started":1}
I20260929 02:09:40.654407 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.624444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.11.182.194:45041)
0929 02:09:40.646082 (+ 21638us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.646092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.646441 (+ 349us) server_negotiation.cc:407] Connection header received
0929 02:09:40.646550 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.646560 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.646768 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.647029 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:40.647775 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.649342 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.651926 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.653297 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.653500 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.653671 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.653898 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.654071 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21465,"thread_start_us":21361,"threads_started":1}
I20260929 02:09:40.655372 27892 tablet_service.cc:1467] Processing CreateTablet for tablet 438f8fb3027a48bdb3355f876cd80e7f (DEFAULT_TABLE table=test-table [id=0e19025bae5a481896ae0979321e1014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:40.655372 27625 tablet_service.cc:1467] Processing CreateTablet for tablet 438f8fb3027a48bdb3355f876cd80e7f (DEFAULT_TABLE table=test-table [id=0e19025bae5a481896ae0979321e1014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:40.656822 27892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438f8fb3027a48bdb3355f876cd80e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:40.656821 27625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438f8fb3027a48bdb3355f876cd80e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:40.657902 27759 tablet_service.cc:1467] Processing CreateTablet for tablet 438f8fb3027a48bdb3355f876cd80e7f (DEFAULT_TABLE table=test-table [id=0e19025bae5a481896ae0979321e1014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:09:40.659111 27759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438f8fb3027a48bdb3355f876cd80e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:09:40.670617 27975 tablet_bootstrap.cc:492] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: Bootstrap starting.
I20260929 02:09:40.673691 27975 tablet_bootstrap.cc:654] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:40.674535 27977 tablet_bootstrap.cc:492] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: Bootstrap starting.
I20260929 02:09:40.675033 27975 log.cc:824] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:40.675416 27976 tablet_bootstrap.cc:492] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: Bootstrap starting.
I20260929 02:09:40.677960 27975 tablet_bootstrap.cc:492] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: No bootstrap required, opened a new log
I20260929 02:09:40.678221 27975 ts_tablet_manager.cc:1397] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 02:09:40.678968 27976 tablet_bootstrap.cc:654] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:40.679615 27977 tablet_bootstrap.cc:654] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: Neither blocks nor log segments found. Creating new log.
I20260929 02:09:40.680195 27976 log.cc:824] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:40.680859 27977 log.cc:824] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:09:40.682695 27976 tablet_bootstrap.cc:492] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: No bootstrap required, opened a new log
I20260929 02:09:40.682996 27976 ts_tablet_manager.cc:1397] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260929 02:09:40.685101 27977 tablet_bootstrap.cc:492] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: No bootstrap required, opened a new log
I20260929 02:09:40.685351 27977 ts_tablet_manager.cc:1397] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260929 02:09:40.686980 27975 raft_consensus.cc:348] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } }
I20260929 02:09:40.687422 27975 raft_consensus.cc:374] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:40.687501 27975 raft_consensus.cc:729] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987f2bc887284e84a9fd63bf4f9483bd, State: Initialized, Role: FOLLOWER
I20260929 02:09:40.688138 27975 consensus_queue.cc:260] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } }
I20260929 02:09:40.689373 27824 heartbeater.cc:503] Master 127.11.182.254:43017 was elected leader, sending a full tablet report...
I20260929 02:09:40.690011 27975 ts_tablet_manager.cc:1428] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 02:09:40.692090 27976 raft_consensus.cc:348] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } }
I20260929 02:09:40.692507 27976 raft_consensus.cc:374] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:40.692613 27976 raft_consensus.cc:729] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5a301f17d8465b93d8b2c392525f04, State: Initialized, Role: FOLLOWER
I20260929 02:09:40.693202 27976 consensus_queue.cc:260] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } }
I20260929 02:09:40.694589 27977 raft_consensus.cc:348] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } }
I20260929 02:09:40.694888 27976 ts_tablet_manager.cc:1428] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260929 02:09:40.695010 27977 raft_consensus.cc:374] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:09:40.695106 27977 raft_consensus.cc:729] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9def730ced4539a20930b721a7372b, State: Initialized, Role: FOLLOWER
I20260929 02:09:40.695735 27977 consensus_queue.cc:260] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } }
I20260929 02:09:40.696756 27957 heartbeater.cc:503] Master 127.11.182.254:43017 was elected leader, sending a full tablet report...
I20260929 02:09:40.697299 27977 ts_tablet_manager.cc:1428] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260929 02:09:40.712693 27981 raft_consensus.cc:482] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:09:40.712941 27981 raft_consensus.cc:504] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } }
I20260929 02:09:40.714428 27981 leader_election.cc:288] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df5a301f17d8465b93d8b2c392525f04 (127.11.182.193:40621), 0a9def730ced4539a20930b721a7372b (127.11.182.195:44701)
I20260929 02:09:40.721642 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.714804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:34409 (local address 127.11.182.193:40621)
0929 02:09:40.715034 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.715053 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.715674 (+ 621us) server_negotiation.cc:407] Connection header received
0929 02:09:40.715957 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.715963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.716054 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.716133 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:40.716989 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.718053 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.719941 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.720734 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.720915 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.720987 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.721271 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.721410 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 02:09:40.721661 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.714918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:59145 (local address 127.11.182.195:44701)
0929 02:09:40.715121 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.715131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.715700 (+ 569us) server_negotiation.cc:407] Connection header received
0929 02:09:40.715957 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.715963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.716035 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.716133 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:09:40.716828 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.717994 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.719980 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.720697 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.720839 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.720924 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.721311 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.721435 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 02:09:40.721781 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.714712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:40621 (local address 127.11.182.194:34409)
0929 02:09:40.715560 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:40.715584 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:40.715815 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:40.716296 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:40.716305 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:40.716336 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:40.716818 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.716830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.718226 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.718229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.719790 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.719804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.720968 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.720974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.721118 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.721148 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:09:40.721218 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 02:09:40.721359 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":5,"spinlock_wait_cycles":17664,"thread_start_us":425,"threads_started":1}
I20260929 02:09:40.722219 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.714809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:44701 (local address 127.11.182.194:59145)
0929 02:09:40.715621 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 02:09:40.715637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:09:40.715785 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:09:40.716166 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:09:40.716170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:09:40.716197 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:09:40.716665 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.716677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.718147 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.718153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.719817 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:09:40.719829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.720966 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.720973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:09:40.721151 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.721173 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:09:40.721275 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 02:09:40.721397 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":24,"thread_start_us":414,"threads_started":1}
I20260929 02:09:40.722810 27645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438f8fb3027a48bdb3355f876cd80e7f" candidate_uuid: "987f2bc887284e84a9fd63bf4f9483bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a301f17d8465b93d8b2c392525f04" is_pre_election: true
I20260929 02:09:40.723286 27912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438f8fb3027a48bdb3355f876cd80e7f" candidate_uuid: "987f2bc887284e84a9fd63bf4f9483bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a9def730ced4539a20930b721a7372b" is_pre_election: true
I20260929 02:09:40.723454 27645 raft_consensus.cc:2454] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 987f2bc887284e84a9fd63bf4f9483bd in term 0.
I20260929 02:09:40.723819 27912 raft_consensus.cc:2454] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 987f2bc887284e84a9fd63bf4f9483bd in term 0.
I20260929 02:09:40.724135 27714 leader_election.cc:302] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 987f2bc887284e84a9fd63bf4f9483bd, df5a301f17d8465b93d8b2c392525f04; no voters:
I20260929 02:09:40.724536 27981 raft_consensus.cc:2781] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:09:40.724676 27981 raft_consensus.cc:482] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:09:40.724788 27981 raft_consensus.cc:3037] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:40.726954 27981 raft_consensus.cc:504] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } }
I20260929 02:09:40.727495 27981 leader_election.cc:288] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [CANDIDATE]: Term 1 election: Requested vote from peers df5a301f17d8465b93d8b2c392525f04 (127.11.182.193:40621), 0a9def730ced4539a20930b721a7372b (127.11.182.195:44701)
I20260929 02:09:40.727902 27645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438f8fb3027a48bdb3355f876cd80e7f" candidate_uuid: "987f2bc887284e84a9fd63bf4f9483bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a301f17d8465b93d8b2c392525f04"
I20260929 02:09:40.727924 27912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438f8fb3027a48bdb3355f876cd80e7f" candidate_uuid: "987f2bc887284e84a9fd63bf4f9483bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a9def730ced4539a20930b721a7372b"
I20260929 02:09:40.728103 27912 raft_consensus.cc:3037] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:40.728109 27645 raft_consensus.cc:3037] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:09:40.730147 27645 raft_consensus.cc:2454] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 987f2bc887284e84a9fd63bf4f9483bd in term 1.
I20260929 02:09:40.730221 27912 raft_consensus.cc:2454] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 987f2bc887284e84a9fd63bf4f9483bd in term 1.
I20260929 02:09:40.730541 27714 leader_election.cc:302] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 987f2bc887284e84a9fd63bf4f9483bd, df5a301f17d8465b93d8b2c392525f04; no voters:
I20260929 02:09:40.730829 27981 raft_consensus.cc:2781] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:09:40.731316 27981 raft_consensus.cc:686] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 1 LEADER]: Becoming Leader. State: Replica: 987f2bc887284e84a9fd63bf4f9483bd, State: Running, Role: LEADER
I20260929 02:09:40.731747 27981 consensus_queue.cc:237] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } }
I20260929 02:09:40.736356 27501 catalog_manager.cc:5495] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 987f2bc887284e84a9fd63bf4f9483bd (127.11.182.194). New cstate: current_term: 1 leader_uuid: "987f2bc887284e84a9fd63bf4f9483bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } health_report { overall_health: HEALTHY } } }
I20260929 02:09:40.771963 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.767136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.11.182.194:45041)
0929 02:09:40.767309 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.767314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.767330 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:09:40.767493 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.767497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.767581 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.767703 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:09:40.768036 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.768763 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.770081 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.770474 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.770566 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.771006 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:09:40.771474 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:09:40.771588 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.771678 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.771740 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:09:40.779629 27912 raft_consensus.cc:1261] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [term 1 FOLLOWER]: Refusing update from remote peer 987f2bc887284e84a9fd63bf4f9483bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:40.779644 27645 raft_consensus.cc:1261] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [term 1 FOLLOWER]: Refusing update from remote peer 987f2bc887284e84a9fd63bf4f9483bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:09:40.780529 27986 consensus_queue.cc:1035] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:40.780843 27981 consensus_queue.cc:1035] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:09:40.811973 27500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:09:40.812297 27500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:09:40.815812 27500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Sending DeleteTablet for 3 replicas of tablet 438f8fb3027a48bdb3355f876cd80e7f
I20260929 02:09:40.816941 27892 tablet_service.cc:1514] Processing DeleteTablet for tablet 438f8fb3027a48bdb3355f876cd80e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:40 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:52124
I20260929 02:09:40.817044 27625 tablet_service.cc:1514] Processing DeleteTablet for tablet 438f8fb3027a48bdb3355f876cd80e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:40 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:59602
I20260929 02:09:40.817154 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 27562
I20260929 02:09:40.818184 27759 tablet_service.cc:1514] Processing DeleteTablet for tablet 438f8fb3027a48bdb3355f876cd80e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:09:40 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:45344
I20260929 02:09:40.826999 28002 tablet_replica.cc:330] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: stopping tablet replica
I20260929 02:09:40.827584 28002 raft_consensus.cc:2229] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:09:40.828358 28002 raft_consensus.cc:2258] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:40.834275 28002 ts_tablet_manager.cc:1905] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:40.834527 27999 tablet_replica.cc:330] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: stopping tablet replica
I20260929 02:09:40.834985 27999 raft_consensus.cc:2229] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:09:40.835491 27999 raft_consensus.cc:2258] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:40.836221 28000 tablet_replica.cc:330] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: stopping tablet replica
I20260929 02:09:40.836603 28000 raft_consensus.cc:2229] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:09:40.837139 28000 raft_consensus.cc:2258] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:09:40.839629 28000 ts_tablet_manager.cc:1905] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:40.840253 27999 ts_tablet_manager.cc:1905] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:09:40.841665 28002 ts_tablet_manager.cc:1918] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:40.841814 28002 log.cc:1196] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/438f8fb3027a48bdb3355f876cd80e7f
Sep 29 02:09:40 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:09:40.842293 28002 ts_tablet_manager.cc:1939] T 438f8fb3027a48bdb3355f876cd80e7f P 987f2bc887284e84a9fd63bf4f9483bd: Deleting consensus metadata
I20260929 02:09:40.844050 27486 catalog_manager.cc:4841] TS 987f2bc887284e84a9fd63bf4f9483bd (127.11.182.194:45041): tablet 438f8fb3027a48bdb3355f876cd80e7f (table test-table [id=0e19025bae5a481896ae0979321e1014]) successfully deleted
I20260929 02:09:40.847116 27999 ts_tablet_manager.cc:1918] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:40.847216 28000 ts_tablet_manager.cc:1918] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:09:40.847272 27999 log.cc:1196] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/438f8fb3027a48bdb3355f876cd80e7f
I20260929 02:09:40.847364 28000 log.cc:1196] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/438f8fb3027a48bdb3355f876cd80e7f
I20260929 02:09:40.847782 27999 ts_tablet_manager.cc:1939] T 438f8fb3027a48bdb3355f876cd80e7f P 0a9def730ced4539a20930b721a7372b: Deleting consensus metadata
I20260929 02:09:40.847860 28000 ts_tablet_manager.cc:1939] T 438f8fb3027a48bdb3355f876cd80e7f P df5a301f17d8465b93d8b2c392525f04: Deleting consensus metadata
I20260929 02:09:40.849581 27488 catalog_manager.cc:4841] TS 0a9def730ced4539a20930b721a7372b (127.11.182.195:44701): tablet 438f8fb3027a48bdb3355f876cd80e7f (table test-table [id=0e19025bae5a481896ae0979321e1014]) successfully deleted
I20260929 02:09:40.849841 27487 catalog_manager.cc:4841] TS df5a301f17d8465b93d8b2c392525f04 (127.11.182.193:40621): tablet 438f8fb3027a48bdb3355f876cd80e7f (table test-table [id=0e19025bae5a481896ae0979321e1014]) successfully deleted
I20260929 02:09:40.851234 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.818383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.11.182.193:40621)
0929 02:09:40.818559 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.818566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.830226 (+ 11660us) server_negotiation.cc:407] Connection header received
0929 02:09:40.830461 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.830467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.830557 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.830665 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:40.832986 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.834176 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.836610 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.837367 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.837465 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.843424 (+ 5959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:40.843437 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:40.843453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:40.843492 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:40.847705 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.848657 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.848665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.848669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.848827 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.849292 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.849300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.849303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.849765 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:40.850078 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.850598 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.850806 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:09:40.851899 27685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:40.940135 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 27562
I20260929 02:09:40.964334 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 27696
Sep 29 02:09:40 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:09:40.983007 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:40.965330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.11.182.194:45041)
0929 02:09:40.965521 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:40.965527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:40.965548 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:09:40.965616 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:40.965622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:40.965719 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:40.965834 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:40.967385 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.968611 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.970428 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:40.971223 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:40.971352 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:40.976816 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:40.976830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:40.976851 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:40.976891 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:40.980117 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.980933 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.980940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.980942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.981073 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:40.981502 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:40.981505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:40.981507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:40.981854 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:40.982102 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:40.982483 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 02:09:40.982659 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:09:40.983735 27819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:41.097031 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 27696
I20260929 02:09:41.115660 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 27829
Sep 29 02:09:41 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:09:41.130270 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:09:41.116423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.11.182.195:44701)
0929 02:09:41.116592 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:09:41.116596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:09:41.116667 (+ 71us) server_negotiation.cc:407] Connection header received
0929 02:09:41.116827 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:09:41.116831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:09:41.116902 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:09:41.116988 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:09:41.118276 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:41.119074 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:41.120450 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:09:41.120817 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:09:41.120914 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:09:41.125135 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:09:41.125149 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:09:41.125164 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:09:41.125198 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:09:41.127861 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:41.128486 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:41.128493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:41.128496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:41.128628 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:09:41.128968 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:09:41.128973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:09:41.128975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:09:41.129353 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:09:41.129598 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 02:09:41.129831 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 02:09:41.130017 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:09:41.131145 27952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:41.221362 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 27829
I20260929 02:09:41.238096 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 27469
I20260929 02:09:41.239285 27531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:09:41.347404 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 27469
I20260929 02:10:01.369387 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:43017
--webserver_interface=127.11.182.254
--webserver_port=38483
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:43017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:01.708065 28007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:01.708379 28007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:01.708478 28007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:01.716665 28007 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:10:01.716777 28007 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:10:01.716827 28007 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:10:01.716876 28007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:10:01.716936 28007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:01.717016 28007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:01.717063 28007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:10:01.717111 28007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:10:01.727118 28007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:43017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:01.728402 28007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:01.729688 28007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:01.753350 28013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:01.753326 28014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:01.753286 28016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:01.753931 28007 server_base.cc:1034] running on GCE node
I20260929 02:10:01.754910 28007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:01.756630 28007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:01.757898 28007 hybrid_clock.cc:648] HybridClock initialized: now 1790647801757824 us; error 62 us; skew 500 ppm
I20260929 02:10:01.758313 28007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:01 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647801, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:01.766021 28007 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:10:01.767999 28007 webserver.cc:460] Webserver started at http://127.11.182.254:38483/ using document root <none> and password file <none>
I20260929 02:10:01.768921 28007 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:01.769104 28007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:01.776304 28007 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:10:01.779794 28023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:01.781265 28007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:10:01.781482 28007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "b541d1ba3284498ba15df4f3d1172d60"
format_stamp: "Formatted at 2026-09-29 02:09:38 on dist-test-slave-q06p"
server_key: "e1a771b9a219b0795d9872a0ac737c1b"
server_key_iv: "595ce0fc216013efcdea1052e3f12834"
server_key_version: "encryptionkey@0"
I20260929 02:10:01.782573 28007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:01.813271 28007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:01.818032 28007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:01.818362 28007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:01.838321 28007 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:43017
I20260929 02:10:01.838348 28074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:43017 every 8 connection(s)
I20260929 02:10:01.840052 28007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:10:01.843550 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 28007
I20260929 02:10:01.844375 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:40621
--local_ip_for_outbound_sockets=127.11.182.193
--tserver_master_addrs=127.11.182.254:43017
--webserver_port=40973
--webserver_interface=127.11.182.193
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:10:01.847486 28075 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:10:01.850811 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Bootstrap starting.
I20260929 02:10:01.880859 28075 log.cc:824] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:01.897131 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:10:01.897905 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Bootstrap complete.
I20260929 02:10:01.907760 28075 raft_consensus.cc:348] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:10:01.908639 28075 raft_consensus.cc:729] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b541d1ba3284498ba15df4f3d1172d60, State: Initialized, Role: FOLLOWER
I20260929 02:10:01.909328 28075 consensus_queue.cc:260] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:10:01.909592 28075 raft_consensus.cc:388] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:10:01.909732 28075 raft_consensus.cc:482] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:10:01.909914 28075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:10:01.913844 28075 raft_consensus.cc:504] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:10:01.914386 28075 leader_election.cc:302] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b541d1ba3284498ba15df4f3d1172d60; no voters:
I20260929 02:10:01.915074 28075 leader_election.cc:288] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:10:01.915266 28079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:10:01.916455 28079 raft_consensus.cc:686] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 2 LEADER]: Becoming Leader. State: Replica: b541d1ba3284498ba15df4f3d1172d60, State: Running, Role: LEADER
I20260929 02:10:01.917054 28079 consensus_queue.cc:237] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:10:01.917234 28075 sys_catalog.cc:564] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:10:01.919555 28081 sys_catalog.cc:455] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b541d1ba3284498ba15df4f3d1172d60. Latest consensus state: current_term: 2 leader_uuid: "b541d1ba3284498ba15df4f3d1172d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } } }
I20260929 02:10:01.919934 28081 sys_catalog.cc:458] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:01.920496 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b541d1ba3284498ba15df4f3d1172d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } } }
I20260929 02:10:01.920791 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:01.929314 28086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:10:01.940119 28086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:10:01.941639 28086 catalog_manager.cc:1259] Loaded cluster ID: e5f5e719a6ef47e2adf8de866030118e
I20260929 02:10:01.941794 28086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:10:01.945959 28086 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:10:01.947567 28086 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Loaded TSK: 0
I20260929 02:10:01.986999 28086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Generated new TSK 1
I20260929 02:10:01.987430 28086 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:10:02.223340 28077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:02.223634 28077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:02.223778 28077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:02.231725 28077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:02.231853 28077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:02.232038 28077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:10:02.242425 28077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:40621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=40973
--tserver_master_addrs=127.11.182.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:02.243896 28077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:02.245216 28077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:02.260342 28104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:02.260950 28103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:02.260993 28106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:02.268678 28077 server_base.cc:1034] running on GCE node
I20260929 02:10:02.269500 28077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:02.270635 28077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:02.271880 28077 hybrid_clock.cc:648] HybridClock initialized: now 1790647802271827 us; error 50 us; skew 500 ppm
I20260929 02:10:02.272236 28077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:02 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:02.276681 28077 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:10:02.278488 28077 webserver.cc:460] Webserver started at http://127.11.182.193:40973/ using document root <none> and password file <none>
I20260929 02:10:02.279466 28077 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:02.279637 28077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:02.285605 28077 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:10:02.288811 28113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:02.290344 28077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:10:02.290520 28077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "df5a301f17d8465b93d8b2c392525f04"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "b9923cf3ab0898074010c409ef3beeb7"
server_key_iv: "362fe4ee734e2204cc690d45e5695492"
server_key_version: "encryptionkey@0"
I20260929 02:10:02.291589 28077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:02.309273 28077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:02.313668 28077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:02.313961 28077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:02.315120 28077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:02.316551 28077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:02.316671 28077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:02.316805 28077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:02.316907 28077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:02.352511 28077 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:40621
I20260929 02:10:02.352618 28225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:40621 every 8 connection(s)
I20260929 02:10:02.354054 28077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:10:02.363886 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 28077
I20260929 02:10:02.364549 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:45041
--local_ip_for_outbound_sockets=127.11.182.194
--tserver_master_addrs=127.11.182.254:43017
--webserver_port=46813
--webserver_interface=127.11.182.194
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:10:02 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:02.381695 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:02.357588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:50297 (local address 127.11.182.254:43017)
0929 02:10:02.358360 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:02.358380 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:02.358926 (+ 546us) server_negotiation.cc:407] Connection header received
0929 02:10:02.361030 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:02.361044 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:02.361380 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:02.361694 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:02.363420 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:02.364858 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:02.367591 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:02.368329 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:02.368506 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:02.373443 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:02.373459 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:02.373477 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:02.373522 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:02.376746 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:02.377609 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:02.377617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:02.377623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:02.377801 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:02.378295 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:02.378305 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:02.378307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:02.378832 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:02.379131 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:02.379932 (+ 801us) server_negotiation.cc:299] Negotiation successful
0929 02:10:02.380333 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":280,"threads_started":1}
I20260929 02:10:02.381695 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:02.357892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:50297)
0929 02:10:02.358729 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:02.358766 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:02.360523 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:02.361858 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:02.361865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:02.362425 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:02.363150 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:02.363170 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:02.365064 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:02.365078 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:02.367413 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:02.367431 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:02.368610 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:02.368620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:02.368991 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:02.369768 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:02.369808 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:02.373096 (+ 3288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:02.376932 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:02.376942 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:02.376967 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:02.377435 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:02.377931 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:02.377936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:02.377940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:02.378157 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:02.379054 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:02.379063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:02.379432 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 02:10:02.379704 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 02:10:02.380308 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":323,"threads_started":1}
I20260929 02:10:02.385036 28226 heartbeater.cc:347] Connected to a master server at 127.11.182.254:43017
I20260929 02:10:02.385391 28226 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:02.386301 28226 heartbeater.cc:511] Master 127.11.182.254:43017 requested a full tablet report, sending...
I20260929 02:10:02.388751 28040 ts_manager.cc:194] Registered new tserver with Master: df5a301f17d8465b93d8b2c392525f04 (127.11.182.193:40621)
I20260929 02:10:02.390267 28040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:50297
W20260929 02:10:02.732457 28230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:02.732801 28230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:02.732963 28230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:02.740393 28230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:02.740557 28230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:02.740739 28230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:10:02.750577 28230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:45041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=46813
--tserver_master_addrs=127.11.182.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:02.751950 28230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:02.753260 28230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:02.766525 28238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:02.766511 28236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:02.766860 28235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:02.767629 28230 server_base.cc:1034] running on GCE node
I20260929 02:10:02.768240 28230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:02.769181 28230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:02.770424 28230 hybrid_clock.cc:648] HybridClock initialized: now 1790647802770372 us; error 34 us; skew 500 ppm
I20260929 02:10:02.770740 28230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:02 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:02.773839 28230 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:10:02.775462 28230 webserver.cc:460] Webserver started at http://127.11.182.194:46813/ using document root <none> and password file <none>
I20260929 02:10:02.776160 28230 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:02.776319 28230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:02.781729 28230 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:10:02.784650 28245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:02.786216 28230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 02:10:02.786463 28230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "987f2bc887284e84a9fd63bf4f9483bd"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "0d0c102ae954fd1898aa0cadd5c26839"
server_key_iv: "fd44a7b52a63271340640e222d2fbfad"
server_key_version: "encryptionkey@0"
I20260929 02:10:02.787686 28230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:02.820916 28230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:02.825397 28230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:02.825690 28230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:02.826705 28230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:02.828087 28230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:02.828213 28230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:02.828351 28230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:02.828439 28230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:02.863011 28230 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:45041
I20260929 02:10:02.863092 28357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:45041 every 8 connection(s)
I20260929 02:10:02.864487 28230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:10:02.873122 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 28230
I20260929 02:10:02.873767 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:44701
--local_ip_for_outbound_sockets=127.11.182.195
--tserver_master_addrs=127.11.182.254:43017
--webserver_port=41945
--webserver_interface=127.11.182.195
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:10:02 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:02.885370 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:02.867064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:56973 (local address 127.11.182.254:43017)
0929 02:10:02.867255 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:02.867260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:02.868474 (+ 1214us) server_negotiation.cc:407] Connection header received
0929 02:10:02.870343 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:02.870349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:02.870470 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:02.870590 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:02.872271 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:02.873015 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:02.874767 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:02.875261 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:02.875432 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:02.879402 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:02.879407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:02.879411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:02.879453 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:02.882013 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:02.882809 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:02.882814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:02.882816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:02.882999 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:02.883478 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:02.883483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:02.883487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:02.883778 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:02.883940 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:02.884780 (+ 840us) server_negotiation.cc:299] Negotiation successful
0929 02:10:02.885020 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:10:02.886483 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:02.867600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:56973)
0929 02:10:02.868300 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:02.868344 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:02.870058 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:02.870889 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:02.870903 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:02.871465 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:02.872074 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:02.872101 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:02.873197 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:02.873203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:02.874614 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:02.874626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:02.875551 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:02.875560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:02.875817 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:02.876690 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:02.876730 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:02.879096 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:02.882191 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:02.882201 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:02.882220 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:02.882662 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:02.883125 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:02.883130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:02.883134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:02.883344 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:02.883914 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:02.883922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:02.884304 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 02:10:02.884608 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 02:10:02.885179 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":278,"threads_started":1}
I20260929 02:10:02.888104 28358 heartbeater.cc:347] Connected to a master server at 127.11.182.254:43017
I20260929 02:10:02.888447 28358 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:02.889356 28358 heartbeater.cc:511] Master 127.11.182.254:43017 requested a full tablet report, sending...
I20260929 02:10:02.891032 28040 ts_manager.cc:194] Registered new tserver with Master: 987f2bc887284e84a9fd63bf4f9483bd (127.11.182.194:45041)
I20260929 02:10:02.891958 28040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:56973
W20260929 02:10:03.212447 28361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:03.212721 28361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:03.212901 28361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:03.220516 28361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:03.220633 28361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:03.220811 28361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:10:03.230798 28361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:44701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=41945
--tserver_master_addrs=127.11.182.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:03.232213 28361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:03.233592 28361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:03.247432 28366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:03.247476 28367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:03.247524 28369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:03.248565 28361 server_base.cc:1034] running on GCE node
I20260929 02:10:03.249245 28361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:03.250298 28361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:03.251552 28361 hybrid_clock.cc:648] HybridClock initialized: now 1790647803251509 us; error 67 us; skew 500 ppm
I20260929 02:10:03.251888 28361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:03 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:03.255509 28361 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:10:03.257035 28361 webserver.cc:460] Webserver started at http://127.11.182.195:41945/ using document root <none> and password file <none>
I20260929 02:10:03.257710 28361 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:03.257824 28361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:03.263486 28361 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:10:03.266463 28376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:03.267935 28361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:10:03.268123 28361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "0a9def730ced4539a20930b721a7372b"
format_stamp: "Formatted at 2026-09-29 02:09:40 on dist-test-slave-q06p"
server_key: "865f09701c31acecba2d9a3444629e9f"
server_key_iv: "4f637059de7e5d3db1a6dda6ef9d6f2e"
server_key_version: "encryptionkey@0"
I20260929 02:10:03.269122 28361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:03.288451 28361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:03.292765 28361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:03.293097 28361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:03.294106 28361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:03.295662 28361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:03.295768 28361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:03.295921 28361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:03.296010 28361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:03.329064 28361 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:44701
I20260929 02:10:03.329141 28488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:44701 every 8 connection(s)
I20260929 02:10:03.330490 28361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:10:03.340456 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 28361
Sep 29 02:10:03 dist-test-slave-q06p krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:03.353121 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.333099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:60829 (local address 127.11.182.254:43017)
0929 02:10:03.333304 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.333309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.334419 (+ 1110us) server_negotiation.cc:407] Connection header received
0929 02:10:03.336389 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.336395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.336503 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.336620 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:03.338138 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.338980 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.340927 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.341606 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.341768 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.346340 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:03.346344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:03.346348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:03.346379 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:03.349435 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.350169 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.350175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.350178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.350304 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.350732 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.350739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.350742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.351160 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:03.351316 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.352214 (+ 898us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.352467 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:10:03.353982 28491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.333543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:60829)
0929 02:10:03.334237 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:03.334286 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:03.336082 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:03.336864 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:03.336872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:03.337389 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:03.337958 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.337976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.339159 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.339163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.340765 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.340793 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.342121 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.342129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.342374 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.343105 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:03.343135 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:03.346131 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:03.349585 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:03.349593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:03.349611 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:03.350040 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:03.350413 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:03.350418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:03.350422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:03.350619 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:03.351296 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:03.351303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:03.351705 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 02:10:03.352062 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 02:10:03.352594 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":268,"threads_started":1}
I20260929 02:10:03.355594 28489 heartbeater.cc:347] Connected to a master server at 127.11.182.254:43017
I20260929 02:10:03.355911 28489 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:03.356825 28489 heartbeater.cc:511] Master 127.11.182.254:43017 requested a full tablet report, sending...
I20260929 02:10:03.358055 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.341517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.11.182.254:43017)
0929 02:10:03.342030 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.342037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.342353 (+ 316us) server_negotiation.cc:407] Connection header received
0929 02:10:03.342542 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.342555 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.342627 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.342735 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:03.344857 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.345858 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.347467 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.348194 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.348336 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.351064 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:03.351070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:03.351075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:03.351105 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:03.354335 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.355302 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.355307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.355310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.355447 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.356349 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.356355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.356358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.356691 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:03.357034 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.357277 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.357496 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":269,"threads_started":1}
I20260929 02:10:03.358470 28040 ts_manager.cc:194] Registered new tserver with Master: 0a9def730ced4539a20930b721a7372b (127.11.182.195:44701)
I20260929 02:10:03.359404 28040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:60829
I20260929 02:10:03.360852 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:10:03.367333 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.361844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.11.182.254:43017)
0929 02:10:03.362016 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.362021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.362643 (+ 622us) server_negotiation.cc:407] Connection header received
0929 02:10:03.362791 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.362794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.363000 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.363112 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:10:03.363531 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.364227 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.365601 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.366011 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.366136 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.366546 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:10:03.366743 (+ 197us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 02:10:03.367114 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48290: token expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 02:10:03.367431 28492 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48290: token expired
W20260929 02:10:03.368026 27962 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.182.254:43017)
I20260929 02:10:03.381101 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.368496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.11.182.254:43017)
0929 02:10:03.368661 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.368667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.368685 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:10:03.368838 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.368844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.368906 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.369009 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:03.370329 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.371110 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.372465 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.372943 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.373082 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.375774 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:03.375778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:03.375782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:03.375812 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:03.378731 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.379408 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.379414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.379416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.379544 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.379879 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.379882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.379885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.380224 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:03.380412 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.380679 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.380841 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:10:03.384810 28039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:10:03.392763 28039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:10:03.393630 28226 heartbeater.cc:503] Master 127.11.182.254:43017 was elected leader, sending a full tablet report...
I20260929 02:10:03.442624 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.405331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:45041 (local address 127.0.0.1:58050)
0929 02:10:03.432689 (+ 27358us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:03.432712 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:03.432846 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:03.434371 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:03.434383 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:03.434419 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:03.435092 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.435111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.437086 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.437094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.439776 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.439790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.441369 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.441377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.441641 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.441811 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 02:10:03.441964 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:10:03.442182 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27229,"mutex_wait_us":86,"thread_start_us":27100,"threads_started":1}
I20260929 02:10:03.442638 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.405483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58050 (local address 127.11.182.194:45041)
0929 02:10:03.433529 (+ 28046us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.433542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.433582 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:10:03.433705 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.433718 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.434080 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.434331 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:03.435323 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.436952 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.439909 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.440861 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.440983 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.441104 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.442079 (+ 975us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.442279 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27809,"thread_start_us":29374,"threads_started":1}
I20260929 02:10:03.445399 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.405015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:44701 (local address 127.0.0.1:47116)
0929 02:10:03.431718 (+ 26703us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:03.431778 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:03.432055 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:03.435795 (+ 3740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:03.435804 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:03.435840 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:03.436419 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.436431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.438313 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.438321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.442136 (+ 3815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.442152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.444514 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.444521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.444694 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.444733 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:10:03.444841 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 02:10:03.444992 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25463,"mutex_wait_us":105,"thread_start_us":31243,"threads_started":1}
I20260929 02:10:03.446247 28500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.405209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.11.182.195:44701)
0929 02:10:03.435199 (+ 29990us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.435205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.435258 (+ 53us) server_negotiation.cc:407] Connection header received
0929 02:10:03.435354 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.435362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.435590 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.435789 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:03.436627 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.438107 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.442330 (+ 4223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.443971 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.444170 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.444341 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.445744 (+ 1403us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.445930 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29866,"thread_start_us":29543,"threads_started":1}
I20260929 02:10:03.446298 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.407151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:40621 (local address 127.0.0.1:60554)
0929 02:10:03.436568 (+ 29417us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:03.436587 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:03.436700 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:03.437287 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:03.437291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:03.437314 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:03.437741 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.437753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.439796 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.439800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.443375 (+ 3575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.443386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.445383 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.445390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.445578 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.445610 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:10:03.445782 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 02:10:03.445921 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29293,"mutex_wait_us":1,"thread_start_us":23730,"threads_started":1}
I20260929 02:10:03.446319 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.407470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.11.182.193:40621)
0929 02:10:03.432331 (+ 24861us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.432345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.436740 (+ 4395us) server_negotiation.cc:407] Connection header received
0929 02:10:03.436855 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.436862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.437087 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.437279 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:03.438106 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.439661 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.443539 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.444926 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.445128 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.445240 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.445823 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.445992 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24633,"thread_start_us":24332,"threads_started":1}
I20260929 02:10:03.447134 28293 tablet_service.cc:1467] Processing CreateTablet for tablet 84ae117be392476194abe392f3f460f9 (DEFAULT_TABLE table=test-table [id=114a13ed3a7947439d9f5c14f119ca25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:03.448616 28293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ae117be392476194abe392f3f460f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:03.450404 28424 tablet_service.cc:1467] Processing CreateTablet for tablet 84ae117be392476194abe392f3f460f9 (DEFAULT_TABLE table=test-table [id=114a13ed3a7947439d9f5c14f119ca25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:03.452000 28424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ae117be392476194abe392f3f460f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:03.453539 28161 tablet_service.cc:1467] Processing CreateTablet for tablet 84ae117be392476194abe392f3f460f9 (DEFAULT_TABLE table=test-table [id=114a13ed3a7947439d9f5c14f119ca25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:03.454704 28161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ae117be392476194abe392f3f460f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:03.464553 28505 tablet_bootstrap.cc:492] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: Bootstrap starting.
I20260929 02:10:03.467031 28506 tablet_bootstrap.cc:492] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: Bootstrap starting.
I20260929 02:10:03.468235 28507 tablet_bootstrap.cc:492] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: Bootstrap starting.
I20260929 02:10:03.468948 28505 tablet_bootstrap.cc:654] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:03.470338 28506 tablet_bootstrap.cc:654] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:03.470347 28505 log.cc:824] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:03.471535 28506 log.cc:824] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:03.471974 28507 tablet_bootstrap.cc:654] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:03.473773 28507 log.cc:824] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:03.476974 28507 tablet_bootstrap.cc:492] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: No bootstrap required, opened a new log
I20260929 02:10:03.477176 28506 tablet_bootstrap.cc:492] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: No bootstrap required, opened a new log
I20260929 02:10:03.477262 28507 ts_tablet_manager.cc:1397] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:10:03.477485 28506 ts_tablet_manager.cc:1397] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260929 02:10:03.479147 28505 tablet_bootstrap.cc:492] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: No bootstrap required, opened a new log
I20260929 02:10:03.479411 28505 ts_tablet_manager.cc:1397] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260929 02:10:03.485144 28506 raft_consensus.cc:348] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } }
I20260929 02:10:03.485785 28506 raft_consensus.cc:374] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:03.485909 28506 raft_consensus.cc:729] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5a301f17d8465b93d8b2c392525f04, State: Initialized, Role: FOLLOWER
I20260929 02:10:03.486686 28506 consensus_queue.cc:260] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } }
I20260929 02:10:03.488173 28507 raft_consensus.cc:348] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } }
I20260929 02:10:03.488615 28506 ts_tablet_manager.cc:1428] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 02:10:03.488674 28507 raft_consensus.cc:374] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:03.488818 28507 raft_consensus.cc:729] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9def730ced4539a20930b721a7372b, State: Initialized, Role: FOLLOWER
I20260929 02:10:03.489470 28507 consensus_queue.cc:260] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } }
I20260929 02:10:03.490038 28505 raft_consensus.cc:348] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } }
I20260929 02:10:03.490557 28505 raft_consensus.cc:374] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:03.490599 28489 heartbeater.cc:503] Master 127.11.182.254:43017 was elected leader, sending a full tablet report...
I20260929 02:10:03.490736 28505 raft_consensus.cc:729] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987f2bc887284e84a9fd63bf4f9483bd, State: Initialized, Role: FOLLOWER
I20260929 02:10:03.491415 28507 ts_tablet_manager.cc:1428] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260929 02:10:03.491562 28505 consensus_queue.cc:260] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } }
I20260929 02:10:03.492765 28358 heartbeater.cc:503] Master 127.11.182.254:43017 was elected leader, sending a full tablet report...
I20260929 02:10:03.494186 28505 ts_tablet_manager.cc:1428] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260929 02:10:03.532102 28513 raft_consensus.cc:482] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:10:03.532490 28513 raft_consensus.cc:504] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } }
I20260929 02:10:03.534134 28513 leader_election.cc:288] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a9def730ced4539a20930b721a7372b (127.11.182.195:44701), df5a301f17d8465b93d8b2c392525f04 (127.11.182.193:40621)
I20260929 02:10:03.543565 28500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.534707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:47167 (local address 127.11.182.195:44701)
0929 02:10:03.534926 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.534934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.535717 (+ 783us) server_negotiation.cc:407] Connection header received
0929 02:10:03.535875 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.535881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.535960 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.536073 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:03.536891 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.538005 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.541075 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.542345 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.542513 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.542591 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.543162 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.543303 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:10:03.543567 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.534603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:59987 (local address 127.11.182.193:40621)
0929 02:10:03.534823 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.534829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.535651 (+ 822us) server_negotiation.cc:407] Connection header received
0929 02:10:03.535875 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.535881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.535960 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.536073 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:03.536891 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.538005 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.541084 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.542341 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.542515 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.542594 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.543162 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.543303 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:10:03.543898 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.534478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:40621 (local address 127.11.182.194:59987)
0929 02:10:03.535509 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:03.535529 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:03.535708 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:03.536106 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:03.536111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:03.536138 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:03.536742 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.536755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.538170 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.538176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.540881 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.540897 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.542639 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.542646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.542898 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.542991 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 02:10:03.543096 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:10:03.543287 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":179,"thread_start_us":700,"threads_started":1}
I20260929 02:10:03.544014 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.534598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:44701 (local address 127.11.182.194:47167)
0929 02:10:03.535612 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:03.535627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:03.535719 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:03.536106 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:03.536111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:03.536199 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:03.536667 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.536686 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.538170 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.538176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.540886 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:03.540897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.542644 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.542649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:03.542938 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.543002 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 02:10:03.543096 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:10:03.543287 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":354,"spinlock_wait_cycles":21888,"thread_start_us":451,"threads_started":1}
I20260929 02:10:03.544975 28181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae117be392476194abe392f3f460f9" candidate_uuid: "987f2bc887284e84a9fd63bf4f9483bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a301f17d8465b93d8b2c392525f04" is_pre_election: true
I20260929 02:10:03.545102 28444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae117be392476194abe392f3f460f9" candidate_uuid: "987f2bc887284e84a9fd63bf4f9483bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a9def730ced4539a20930b721a7372b" is_pre_election: true
I20260929 02:10:03.545531 28181 raft_consensus.cc:2454] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 987f2bc887284e84a9fd63bf4f9483bd in term 0.
I20260929 02:10:03.545639 28444 raft_consensus.cc:2454] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 987f2bc887284e84a9fd63bf4f9483bd in term 0.
I20260929 02:10:03.546257 28248 leader_election.cc:302] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 987f2bc887284e84a9fd63bf4f9483bd, df5a301f17d8465b93d8b2c392525f04; no voters:
I20260929 02:10:03.546689 28513 raft_consensus.cc:2781] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:10:03.546883 28513 raft_consensus.cc:482] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:10:03.547009 28513 raft_consensus.cc:3037] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:03.549166 28513 raft_consensus.cc:504] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } }
I20260929 02:10:03.549809 28513 leader_election.cc:288] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [CANDIDATE]: Term 1 election: Requested vote from peers 0a9def730ced4539a20930b721a7372b (127.11.182.195:44701), df5a301f17d8465b93d8b2c392525f04 (127.11.182.193:40621)
I20260929 02:10:03.550233 28444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae117be392476194abe392f3f460f9" candidate_uuid: "987f2bc887284e84a9fd63bf4f9483bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a9def730ced4539a20930b721a7372b"
I20260929 02:10:03.550354 28181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae117be392476194abe392f3f460f9" candidate_uuid: "987f2bc887284e84a9fd63bf4f9483bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a301f17d8465b93d8b2c392525f04"
I20260929 02:10:03.550463 28444 raft_consensus.cc:3037] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:03.550606 28181 raft_consensus.cc:3037] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:03.552827 28444 raft_consensus.cc:2454] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 987f2bc887284e84a9fd63bf4f9483bd in term 1.
I20260929 02:10:03.552839 28181 raft_consensus.cc:2454] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 987f2bc887284e84a9fd63bf4f9483bd in term 1.
I20260929 02:10:03.553284 28249 leader_election.cc:302] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a9def730ced4539a20930b721a7372b, 987f2bc887284e84a9fd63bf4f9483bd; no voters:
I20260929 02:10:03.553663 28513 raft_consensus.cc:2781] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:03.554090 28513 raft_consensus.cc:686] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 1 LEADER]: Becoming Leader. State: Replica: 987f2bc887284e84a9fd63bf4f9483bd, State: Running, Role: LEADER
I20260929 02:10:03.554574 28513 consensus_queue.cc:237] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } }
I20260929 02:10:03.559432 28038 catalog_manager.cc:5495] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 987f2bc887284e84a9fd63bf4f9483bd (127.11.182.194). New cstate: current_term: 1 leader_uuid: "987f2bc887284e84a9fd63bf4f9483bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "987f2bc887284e84a9fd63bf4f9483bd" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 } health_report { overall_health: UNKNOWN } } }
W20260929 02:10:03.582160 28490 tablet.cc:2369] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:03.588605 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.582891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.11.182.194:45041)
0929 02:10:03.583109 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.583113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.583187 (+ 74us) server_negotiation.cc:407] Connection header received
0929 02:10:03.583384 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.583388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.583467 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.583590 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:10:03.583960 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.584843 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.586405 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.586814 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.587025 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.587541 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:10:03.588097 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:10:03.588226 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.588318 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.588393 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:10:03.597915 28181 raft_consensus.cc:1261] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [term 1 FOLLOWER]: Refusing update from remote peer 987f2bc887284e84a9fd63bf4f9483bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:03.597916 28444 raft_consensus.cc:1261] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [term 1 FOLLOWER]: Refusing update from remote peer 987f2bc887284e84a9fd63bf4f9483bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:03.599215 28516 consensus_queue.cc:1035] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5a301f17d8465b93d8b2c392525f04" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:03.599596 28513 consensus_queue.cc:1035] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a9def730ced4539a20930b721a7372b" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:10:03.606997 28227 tablet.cc:2369] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:03.632467 28038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:03.632840 28038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:03.636435 28038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Sending DeleteTablet for 3 replicas of tablet 84ae117be392476194abe392f3f460f9
I20260929 02:10:03.637687 28161 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ae117be392476194abe392f3f460f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:03 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:60554
I20260929 02:10:03.637684 28424 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ae117be392476194abe392f3f460f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:03 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:47116
I20260929 02:10:03.638525 28293 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ae117be392476194abe392f3f460f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:03 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:58050
I20260929 02:10:03.638707 28527 tablet_replica.cc:330] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: stopping tablet replica
I20260929 02:10:03.639202 28527 raft_consensus.cc:2229] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:03.639638 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 28077
I20260929 02:10:03.639768 28527 raft_consensus.cc:2258] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:03.641100 28528 tablet_replica.cc:330] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: stopping tablet replica
I20260929 02:10:03.641834 28528 raft_consensus.cc:2229] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:03.643951 28527 ts_tablet_manager.cc:1905] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:03.644326 28528 raft_consensus.cc:2258] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:03.645490 28529 tablet_replica.cc:330] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: stopping tablet replica
I20260929 02:10:03.645898 28529 raft_consensus.cc:2229] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:10:03.646555 28529 raft_consensus.cc:2258] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:03.646579 28528 ts_tablet_manager.cc:1905] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:03.649709 28529 ts_tablet_manager.cc:1905] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:03.651654 28527 ts_tablet_manager.cc:1918] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:03.651863 28527 log.cc:1196] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/84ae117be392476194abe392f3f460f9
I20260929 02:10:03.652484 28527 ts_tablet_manager.cc:1939] T 84ae117be392476194abe392f3f460f9 P df5a301f17d8465b93d8b2c392525f04: Deleting consensus metadata
I20260929 02:10:03.654559 28026 catalog_manager.cc:4841] TS df5a301f17d8465b93d8b2c392525f04 (127.11.182.193:40621): tablet 84ae117be392476194abe392f3f460f9 (table test-table [id=114a13ed3a7947439d9f5c14f119ca25]) successfully deleted
I20260929 02:10:03.654644 28528 ts_tablet_manager.cc:1918] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:03.654805 28528 log.cc:1196] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/84ae117be392476194abe392f3f460f9
I20260929 02:10:03.655532 28528 ts_tablet_manager.cc:1939] T 84ae117be392476194abe392f3f460f9 P 0a9def730ced4539a20930b721a7372b: Deleting consensus metadata
I20260929 02:10:03.656775 28529 ts_tablet_manager.cc:1918] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:03.656980 28529 log.cc:1196] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/84ae117be392476194abe392f3f460f9
I20260929 02:10:03.657546 28027 catalog_manager.cc:4841] TS 0a9def730ced4539a20930b721a7372b (127.11.182.195:44701): tablet 84ae117be392476194abe392f3f460f9 (table test-table [id=114a13ed3a7947439d9f5c14f119ca25]) successfully deleted
I20260929 02:10:03.657621 28529 ts_tablet_manager.cc:1939] T 84ae117be392476194abe392f3f460f9 P 987f2bc887284e84a9fd63bf4f9483bd: Deleting consensus metadata
I20260929 02:10:03.658866 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.640601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.11.182.193:40621)
0929 02:10:03.640793 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.640801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.640900 (+ 99us) server_negotiation.cc:407] Connection header received
0929 02:10:03.641133 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.641139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.641229 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.641343 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:03.643683 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.644725 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.646766 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.647462 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.647598 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.650980 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:03.650996 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:03.651013 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:03.651055 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:03.655208 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.656188 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.656196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.656200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.656352 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.656840 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.656845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.656848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.657312 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:03.657606 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.658199 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.658434 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:10:03.659394 28025 catalog_manager.cc:4841] TS 987f2bc887284e84a9fd63bf4f9483bd (127.11.182.194:45041): tablet 84ae117be392476194abe392f3f460f9 (table test-table [id=114a13ed3a7947439d9f5c14f119ca25]) successfully deleted
I20260929 02:10:03.659608 28221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:03.751586 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 28077
I20260929 02:10:03.776922 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 28230
I20260929 02:10:03.794472 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.777806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.11.182.194:45041)
0929 02:10:03.778012 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.778017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.778037 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:10:03.778264 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.778269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.778344 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.778438 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:03.780204 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.781032 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.783001 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.783500 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.783619 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.786710 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:03.786731 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:03.786753 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:03.786805 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:03.791274 (+ 4469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.792117 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.792127 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.792131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.792327 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.792728 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.792733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.792737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.793213 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:03.793564 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.793832 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.794061 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 02:10:03.795210 28353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:03.908468 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 28230
I20260929 02:10:03.928087 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 28361
I20260929 02:10:03.941321 28500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:03.928837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.11.182.195:44701)
0929 02:10:03.929044 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:03.929052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:03.929095 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:10:03.929318 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:03.929322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:03.929384 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:03.929504 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:03.930881 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.931698 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.933073 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:03.933474 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:03.933602 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:03.936096 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:03.936110 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:03.936126 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:03.936171 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:03.938650 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.939344 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.939359 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.939363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.939485 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:03.939833 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:03.939839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:03.939843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:03.940187 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:03.940476 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:03.940774 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 02:10:03.941010 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:10:03.941954 28484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:04.030483 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 28361
I20260929 02:10:04.047183 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 28007
I20260929 02:10:04.048332 28070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:04.161670 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 28007
I20260929 02:10:24.183494 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:43017
--webserver_interface=127.11.182.254
--webserver_port=38483
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:43017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:24.511421 28534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:24.511739 28534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:24.511911 28534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:24.518973 28534 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:10:24.519090 28534 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:10:24.519160 28534 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:10:24.519235 28534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:10:24.519315 28534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:24.519416 28534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:24.519487 28534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:10:24.519567 28534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:10:24.529212 28534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:43017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:24.530596 28534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:24.531934 28534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:24.544283 28539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:24.544435 28540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:24.544509 28534 server_base.cc:1034] running on GCE node
W20260929 02:10:24.544411 28542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:24.545565 28534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:24.547091 28534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:24.548311 28534 hybrid_clock.cc:648] HybridClock initialized: now 1790647824548229 us; error 50 us; skew 500 ppm
I20260929 02:10:24.548640 28534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:24 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647824, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:24.552248 28534 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:10:24.553757 28534 webserver.cc:460] Webserver started at http://127.11.182.254:38483/ using document root <none> and password file <none>
I20260929 02:10:24.554520 28534 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:24.554649 28534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:24.560580 28534 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:10:24.563647 28549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:24.565023 28534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:10:24.565212 28534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "b541d1ba3284498ba15df4f3d1172d60"
format_stamp: "Formatted at 2026-09-29 02:09:38 on dist-test-slave-q06p"
server_key: "e1a771b9a219b0795d9872a0ac737c1b"
server_key_iv: "595ce0fc216013efcdea1052e3f12834"
server_key_version: "encryptionkey@0"
I20260929 02:10:24.566293 28534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:24.595796 28534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:24.600863 28534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:24.601194 28534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:24.621037 28534 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:43017
I20260929 02:10:24.621066 28600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:43017 every 8 connection(s)
I20260929 02:10:24.622622 28534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:10:24.627542 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 28534
I20260929 02:10:24.628505 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:40621
--local_ip_for_outbound_sockets=127.11.182.193
--tserver_master_addrs=127.11.182.254:43017
--webserver_port=40973
--webserver_interface=127.11.182.193
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:10:24.630358 28601 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:10:24.635623 28601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Bootstrap starting.
I20260929 02:10:24.656986 28601 log.cc:824] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:24.673072 28601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:10:24.673623 28601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Bootstrap complete.
I20260929 02:10:24.682696 28601 raft_consensus.cc:348] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:10:24.683683 28601 raft_consensus.cc:729] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b541d1ba3284498ba15df4f3d1172d60, State: Initialized, Role: FOLLOWER
I20260929 02:10:24.684332 28601 consensus_queue.cc:260] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:10:24.684667 28601 raft_consensus.cc:388] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:10:24.684792 28601 raft_consensus.cc:482] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:10:24.684975 28601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:10:24.688717 28601 raft_consensus.cc:504] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:10:24.689175 28601 leader_election.cc:302] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b541d1ba3284498ba15df4f3d1172d60; no voters:
I20260929 02:10:24.689831 28601 leader_election.cc:288] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:10:24.689976 28605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:10:24.692488 28601 sys_catalog.cc:564] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:10:24.697167 28605 raft_consensus.cc:686] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [term 3 LEADER]: Becoming Leader. State: Replica: b541d1ba3284498ba15df4f3d1172d60, State: Running, Role: LEADER
I20260929 02:10:24.697852 28605 consensus_queue.cc:237] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } }
I20260929 02:10:24.702000 28606 sys_catalog.cc:455] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b541d1ba3284498ba15df4f3d1172d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } } }
I20260929 02:10:24.702445 28606 sys_catalog.cc:458] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:24.702376 28607 sys_catalog.cc:455] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b541d1ba3284498ba15df4f3d1172d60. Latest consensus state: current_term: 3 leader_uuid: "b541d1ba3284498ba15df4f3d1172d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541d1ba3284498ba15df4f3d1172d60" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 43017 } } }
I20260929 02:10:24.702766 28607 sys_catalog.cc:458] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60 [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:24.708491 28617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:10:24.717093 28617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:10:24.718652 28617 catalog_manager.cc:1259] Loaded cluster ID: e5f5e719a6ef47e2adf8de866030118e
I20260929 02:10:24.718750 28617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:10:24.722784 28617 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:10:24.724304 28617 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Loaded TSK: 0
I20260929 02:10:24.724448 28617 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b541d1ba3284498ba15df4f3d1172d60: Loaded TSK: 1
F20260929 02:10:24.725558 28617 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb8dcca59ad google::LogMessage::Fail() at ??:0
@ 0x7fb8dcca888a google::LogMessage::SendToLog() at ??:0
@ 0x7fb8dcca54de google::LogMessage::Flush() at ??:0
@ 0x7fb8dcca6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb8e0a097d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb8e09f873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb8f26d41c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb8f26d3783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb8f275f2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb8f275f0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb8f27a95e9 std::function<>::operator()() at ??:0
@ 0x7fb8f26d7cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb8f26d5c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb8f275d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb8f275d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb8df8f1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb8df8f491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb8df8f4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb8df8cfeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb8e2ba96db start_thread at ??:0
@ 0x7fb8dbb3f61f clone at ??:0
W20260929 02:10:25.012312 28603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:25.012598 28603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:25.012763 28603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:25.020036 28603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:25.020180 28603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:25.020387 28603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:10:25.030334 28603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:40621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=40973
--tserver_master_addrs=127.11.182.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:25.031669 28603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:25.032995 28603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:25.045868 28650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:25.045722 28652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:25.045967 28649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:25.047375 28603 server_base.cc:1034] running on GCE node
I20260929 02:10:25.047904 28603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:25.048722 28603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:25.049949 28603 hybrid_clock.cc:648] HybridClock initialized: now 1790647825049903 us; error 34 us; skew 500 ppm
I20260929 02:10:25.050207 28603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:25 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:25.053184 28603 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:10:25.054576 28603 webserver.cc:460] Webserver started at http://127.11.182.193:40973/ using document root <none> and password file <none>
I20260929 02:10:25.055258 28603 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:25.055358 28603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:25.060400 28603 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:10:25.062934 28659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:25.064291 28603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:10:25.064471 28603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "df5a301f17d8465b93d8b2c392525f04"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "b9923cf3ab0898074010c409ef3beeb7"
server_key_iv: "362fe4ee734e2204cc690d45e5695492"
server_key_version: "encryptionkey@0"
I20260929 02:10:25.065435 28603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:25.086072 28603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:25.090085 28603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:25.090375 28603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:25.091403 28603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:25.092785 28603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:25.092882 28603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:25.093048 28603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:25.093148 28603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:25.127642 28603 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:40621
I20260929 02:10:25.127772 28771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:40621 every 8 connection(s)
I20260929 02:10:25.129119 28603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:10:25.134089 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:25.132114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:36223)
0929 02:10:25.132919 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:25.133189 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":282,"threads_started":1}
W20260929 02:10:25.134758 28772 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:43017 (0 consecutive failures): Network error: Failed to ping master at 127.11.182.254:43017: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
I20260929 02:10:25.135654 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:25.135317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:53867)
0929 02:10:25.135474 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:25.135545 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:10:25.135859 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 28603
I20260929 02:10:25.136610 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:45041
--local_ip_for_outbound_sockets=127.11.182.194
--tserver_master_addrs=127.11.182.254:43017
--webserver_port=46813
--webserver_interface=127.11.182.194
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:10:25.136663 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:25.136346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:35675)
0929 02:10:25.136477 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:25.136550 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:10:25.137048 28772 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:10:25.462064 28775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:25.462338 28775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:25.462498 28775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:25.469904 28775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:25.470029 28775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:25.470216 28775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:10:25.479920 28775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:45041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=46813
--tserver_master_addrs=127.11.182.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:25.481166 28775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:25.482427 28775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:25.495677 28783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:25.495812 28780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:25.495754 28781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:25.496598 28775 server_base.cc:1034] running on GCE node
I20260929 02:10:25.497267 28775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:25.498241 28775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:25.499500 28775 hybrid_clock.cc:648] HybridClock initialized: now 1790647825499457 us; error 34 us; skew 500 ppm
I20260929 02:10:25.499842 28775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:25 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:25.502935 28775 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:10:25.504288 28775 webserver.cc:460] Webserver started at http://127.11.182.194:46813/ using document root <none> and password file <none>
I20260929 02:10:25.505021 28775 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:25.505168 28775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:25.510300 28775 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:10:25.512985 28790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:25.514266 28775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:10:25.514446 28775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "987f2bc887284e84a9fd63bf4f9483bd"
format_stamp: "Formatted at 2026-09-29 02:09:39 on dist-test-slave-q06p"
server_key: "0d0c102ae954fd1898aa0cadd5c26839"
server_key_iv: "fd44a7b52a63271340640e222d2fbfad"
server_key_version: "encryptionkey@0"
I20260929 02:10:25.515488 28775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:25.538545 28775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:25.542763 28775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:25.543056 28775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:25.544005 28775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:25.545324 28775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:25.545449 28775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:25.545583 28775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:25.545668 28775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:25.578549 28775 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:45041
I20260929 02:10:25.578636 28902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:45041 every 8 connection(s)
I20260929 02:10:25.580067 28775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:10:25.584946 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:25.583028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:54489)
0929 02:10:25.583748 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:25.584017 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":261,"threads_started":1}
W20260929 02:10:25.585579 28903 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:43017 (0 consecutive failures): Network error: Failed to ping master at 127.11.182.254:43017: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
I20260929 02:10:25.586426 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:25.586112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:40477)
0929 02:10:25.586253 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:25.586321 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 02:10:25.587296 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:25.586992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:40409)
0929 02:10:25.587105 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:25.587182 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 02:10:25.587540 28903 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:10:25.589900 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 28775
I20260929 02:10:25.590551 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:44701
--local_ip_for_outbound_sockets=127.11.182.195
--tserver_master_addrs=127.11.182.254:43017
--webserver_port=41945
--webserver_interface=127.11.182.195
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:25.912892 28906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:25.913146 28906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:25.913236 28906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:25.920346 28906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:25.920446 28906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:25.920553 28906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:10:25.929832 28906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:44701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=41945
--tserver_master_addrs=127.11.182.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:25.930997 28906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:25.932339 28906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:25.944931 28911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:25.944967 28914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:25.945118 28912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:25.946561 28906 server_base.cc:1034] running on GCE node
I20260929 02:10:25.947135 28906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:25.947952 28906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:25.949151 28906 hybrid_clock.cc:648] HybridClock initialized: now 1790647825949112 us; error 38 us; skew 500 ppm
I20260929 02:10:25.949451 28906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:25 dist-test-slave-q06p krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:25.952539 28906 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:10:25.953960 28906 webserver.cc:460] Webserver started at http://127.11.182.195:41945/ using document root <none> and password file <none>
I20260929 02:10:25.954617 28906 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:25.954761 28906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:25.960265 28906 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 02:10:25.962829 28921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:25.964110 28906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:10:25.964283 28906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "0a9def730ced4539a20930b721a7372b"
format_stamp: "Formatted at 2026-09-29 02:09:40 on dist-test-slave-q06p"
server_key: "865f09701c31acecba2d9a3444629e9f"
server_key_iv: "4f637059de7e5d3db1a6dda6ef9d6f2e"
server_key_version: "encryptionkey@0"
I20260929 02:10:25.965283 28906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:26.004444 28906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:26.008350 28906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:26.008618 28906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:26.009598 28906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:26.010967 28906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:26.011096 28906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:26.011229 28906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:26.011317 28906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:26.044539 28906 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:44701
I20260929 02:10:26.044627 29033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:44701 every 8 connection(s)
I20260929 02:10:26.045972 28906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:10:26.046676 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 28906
I20260929 02:10:26.051740 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:26.049932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:51475)
0929 02:10:26.050632 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:26.050916 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":268,"threads_started":1}
W20260929 02:10:26.052415 29034 heartbeater.cc:644] Failed to heartbeat to 127.11.182.254:43017 (0 consecutive failures): Network error: Failed to ping master at 127.11.182.254:43017: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
I20260929 02:10:26.053136 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:26.052844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:58977)
0929 02:10:26.052974 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:26.053037 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:10:26.053931 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:26.053616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:53879)
0929 02:10:26.053750 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:26.053827 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:10:26.054155 29034 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:10:26.138242 29038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:26.137502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:38483)
0929 02:10:26.137941 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:26.138024 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":200,"threads_started":1}
I20260929 02:10:26.588805 29039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:26.588023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:53733)
0929 02:10:26.588487 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:26.588573 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":230,"threads_started":1}
I20260929 02:10:27.055631 29040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:27.054758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:58485)
0929 02:10:27.055293 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:27.055408 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":303,"threads_started":1}
I20260929 02:10:27.139930 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:27.139180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:34913)
0929 02:10:27.139652 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:27.139734 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260929 02:10:27.590445 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:27.589713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:34239)
0929 02:10:27.590185 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:27.590264 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
I20260929 02:10:28.057116 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:28.056460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:41453)
0929 02:10:28.056874 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:28.056955 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":215,"threads_started":1}
I20260929 02:10:28.141479 29044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:28.140783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:52025)
0929 02:10:28.141225 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:28.141305 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260929 02:10:28.592142 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:28.591344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:40509)
0929 02:10:28.591867 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:28.591953 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":303,"threads_started":1}
I20260929 02:10:29.058673 29046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:29.058014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:50445)
0929 02:10:29.058428 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:29.058510 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":214,"threads_started":1}
I20260929 02:10:29.143150 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:29.142366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:43507)
0929 02:10:29.142814 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:29.142959 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
I20260929 02:10:29.593873 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:29.593047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:57067)
0929 02:10:29.593575 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:29.593658 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":302,"threads_started":1}
I20260929 02:10:30.060240 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:30.059558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:36311)
0929 02:10:30.059992 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:30.060071 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260929 02:10:30.144796 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:30.144056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:36511)
0929 02:10:30.144509 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:30.144605 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260929 02:10:30.595620 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:30.594934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:43419)
0929 02:10:30.595360 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:30.595439 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":213,"threads_started":1}
I20260929 02:10:31.061801 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:31.061126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:58425)
0929 02:10:31.061553 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:31.061634 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":227,"threads_started":1}
I20260929 02:10:31.146325 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:31.145660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:40951)
0929 02:10:31.146085 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:31.146172 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":234,"threads_started":1}
I20260929 02:10:31.597195 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:31.596531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:59859)
0929 02:10:31.596959 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:31.597043 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":223,"threads_started":1}
I20260929 02:10:32.063495 29055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:32.062710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:47051)
0929 02:10:32.063222 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:32.063311 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":290,"threads_started":1}
I20260929 02:10:32.148073 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:32.147332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:53813)
0929 02:10:32.147781 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:32.147866 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":200,"threads_started":1}
I20260929 02:10:32.598891 29057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:32.598088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:46915)
0929 02:10:32.598564 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:32.598641 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":222,"threads_started":1}
I20260929 02:10:33.065263 29058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:33.064422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:44359)
0929 02:10:33.064976 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:33.065060 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":282,"threads_started":1}
I20260929 02:10:33.149907 29059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:33.148937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:57723)
0929 02:10:33.149534 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:33.149651 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":310,"threads_started":1}
I20260929 02:10:33.600840 29060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:33.599901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:51043)
0929 02:10:33.600491 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:33.600604 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":288,"threads_started":1}
I20260929 02:10:34.066740 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:34.066060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:45813)
0929 02:10:34.066474 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:34.066553 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260929 02:10:34.151549 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:34.150786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:34865)
0929 02:10:34.151301 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:34.151377 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":276,"threads_started":1}
I20260929 02:10:34.602599 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:34.601773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:42429)
0929 02:10:34.602279 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:34.602362 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":252,"threads_started":1}
I20260929 02:10:35.068447 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:35.067598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:52517)
0929 02:10:35.068097 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:35.068188 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260929 02:10:35.153105 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:35.152435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:41419)
0929 02:10:35.152864 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:35.152944 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":235,"threads_started":1}
I20260929 02:10:35.604324 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:35.603563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:51899)
0929 02:10:35.604080 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:35.604161 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":308,"threads_started":1}
I20260929 02:10:36.070072 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:36.069389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:46005)
0929 02:10:36.069837 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:36.069914 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":212,"threads_started":1}
I20260929 02:10:36.154690 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:36.153999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:46969)
0929 02:10:36.154411 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:36.154506 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":211,"threads_started":1}
I20260929 02:10:36.605917 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:36.605224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:41885)
0929 02:10:36.605650 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:36.605743 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260929 02:10:37.071646 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:37.070955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:37719)
0929 02:10:37.071376 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:37.071466 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260929 02:10:37.156538 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:37.155780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:34165)
0929 02:10:37.156290 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:37.156388 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":289,"threads_started":1}
I20260929 02:10:37.607604 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:37.606946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:36689)
0929 02:10:37.607370 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:37.607451 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":215,"threads_started":1}
I20260929 02:10:38.073115 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:38.072445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:53833)
0929 02:10:38.072885 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:38.072963 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":240,"threads_started":1}
I20260929 02:10:38.158213 29074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:38.157472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:57837)
0929 02:10:38.157941 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:38.158031 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":234,"threads_started":1}
I20260929 02:10:38.609571 29075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:38.608655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:39183)
0929 02:10:38.609267 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:38.609380 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":306,"threads_started":1}
I20260929 02:10:39.074666 29076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:39.073960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:48107)
0929 02:10:39.074392 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:39.074483 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":222,"threads_started":1}
I20260929 02:10:39.159960 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:39.159083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:53737)
0929 02:10:39.159594 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:39.159732 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
I20260929 02:10:39.611227 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:39.610450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:58571)
0929 02:10:39.610956 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:39.611036 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260929 02:10:40.076141 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:40.075472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:53047)
0929 02:10:40.075887 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:40.075964 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":202,"threads_started":1}
I20260929 02:10:40.161753 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:40.160994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.193:56083)
0929 02:10:40.161438 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:40.161561 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":224,"threads_started":1}
I20260929 02:10:40.613011 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:40.612296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.194:57637)
0929 02:10:40.612727 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:40.612835 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":240,"threads_started":1}
I20260929 02:10:41.053267 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 28603
I20260929 02:10:41.076117 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:41.054356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.11.182.193:40621)
0929 02:10:41.054924 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:41.054939 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:41.054974 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:10:41.056478 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:41.056488 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:41.056902 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:41.057191 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:41.058504 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:41.059870 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:41.061041 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:41.061681 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:41.061814 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:41.070411 (+ 8597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:41.070419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:41.070434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:41.070475 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:41.073149 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:41.073773 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:41.073779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:41.073786 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:41.073945 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:41.074336 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:41.074339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:41.074341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:41.074833 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:41.075161 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:41.075521 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 02:10:41.075845 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260929 02:10:41.077641 28767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:41.077867 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:41.077104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:43017 (local address 127.11.182.195:33487)
0929 02:10:41.077537 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:41.077628 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":243,"threads_started":1}
I20260929 02:10:41.160914 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 28603
I20260929 02:10:41.179141 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 28775
I20260929 02:10:41.196022 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:41.180099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.11.182.194:45041)
0929 02:10:41.180611 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:41.180624 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:41.180662 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:10:41.182123 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:41.182131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:41.182546 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:41.182932 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:41.184180 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:41.185377 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:41.186695 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:41.187461 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:41.187597 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:41.190152 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:41.190159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:41.190173 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:41.190211 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:41.193041 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:41.193735 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:41.193741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:41.193748 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:41.193916 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:41.194252 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:41.194255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:41.194257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:41.194742 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:41.195045 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:41.195384 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 02:10:41.195707 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":238,"threads_started":1}
I20260929 02:10:41.197460 28898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:41.280926 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 28775
I20260929 02:10:41.298031 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 28906
I20260929 02:10:41.315511 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:41.299088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.11.182.195:44701)
0929 02:10:41.299731 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:41.299746 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:41.299787 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:10:41.301379 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:41.301388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:41.301805 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:41.302095 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:41.303612 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:41.304832 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:41.306114 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:41.306766 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:41.306959 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:41.309445 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:41.309452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:41.309467 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:41.309513 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:41.312456 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:41.313082 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:41.313088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:41.313094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:41.313253 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:41.313626 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:41.313630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:41.313632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:41.314121 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:41.314361 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:41.314769 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 02:10:41.315208 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":323,"threads_started":1}
I20260929 02:10:41.316844 29029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:41.399013 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 28906
2026-09-29T02:10:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64205 ms)
[----------] 30 tests from SecurityITest (301184 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:10:41 dist-test-slave-q06p krb5kdc[29091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:10:41 dist-test-slave-q06p krb5kdc[29091](info): set up 2 sockets
Sep 29 02:10:41 dist-test-slave-q06p krb5kdc[29091](info): commencing operation
krb5kdc: starting...
W20260929 02:10:42.499583 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:10:42 dist-test-slave-q06p krb5kdc[29091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:10:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:10:42.738444 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:39203
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:39203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:43.065557 29104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:43.065834 29104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:43.066001 29104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:43.073239 29104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:10:43.073364 29104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:43.073447 29104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:43.073549 29104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:10:43.073632 29104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:10:43.083213 29104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:39203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:39203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:43.084481 29104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:43.085762 29104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:43.097105 29109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:43.097074 29110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:43.097745 29112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:43.097885 29104 server_base.cc:1034] running on GCE node
I20260929 02:10:43.098697 29104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:43.100061 29104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:43.101259 29104 hybrid_clock.cc:648] HybridClock initialized: now 1790647843101196 us; error 53 us; skew 500 ppm
I20260929 02:10:43.101593 29104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:43 dist-test-slave-q06p krb5kdc[29091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647843, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:43.104566 29104 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:10:43.105901 29104 webserver.cc:460] Webserver started at http://127.11.182.254:40713/ using document root <none> and password file <none>
I20260929 02:10:43.106557 29104 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:43.106700 29104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:43.107051 29104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:43.109861 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "b2763ae3d60c47e6b70cfec559cf55df"
format_stamp: "Formatted at 2026-09-29 02:10:43 on dist-test-slave-q06p"
server_key: "0a56c7cbde053d2e040759868f94d5ba"
server_key_iv: "ec6ebdeac254bb7aa125b7c03fad56fa"
server_key_version: "encryptionkey@0"
I20260929 02:10:43.110577 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "b2763ae3d60c47e6b70cfec559cf55df"
format_stamp: "Formatted at 2026-09-29 02:10:43 on dist-test-slave-q06p"
server_key: "0a56c7cbde053d2e040759868f94d5ba"
server_key_iv: "ec6ebdeac254bb7aa125b7c03fad56fa"
server_key_version: "encryptionkey@0"
I20260929 02:10:43.115505 29104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:10:43.118741 29119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:43.120162 29104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:10:43.120363 29104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "b2763ae3d60c47e6b70cfec559cf55df"
format_stamp: "Formatted at 2026-09-29 02:10:43 on dist-test-slave-q06p"
server_key: "0a56c7cbde053d2e040759868f94d5ba"
server_key_iv: "ec6ebdeac254bb7aa125b7c03fad56fa"
server_key_version: "encryptionkey@0"
I20260929 02:10:43.120581 29104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:43.147394 29104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:43.151496 29104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:43.151811 29104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:43.171247 29104 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:39203
I20260929 02:10:43.171267 29170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:39203 every 8 connection(s)
I20260929 02:10:43.172832 29104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:10:43.176492 29171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:43.178926 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 29104
I20260929 02:10:43.179137 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:10:43.179920 11995 external_mini_cluster.cc:1428] Setting key 207cede1f42f17042e2d73aca5beff90
I20260929 02:10:43.189046 29171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df: Bootstrap starting.
I20260929 02:10:43.192788 29171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:43.194072 29171 log.cc:824] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:10:43 dist-test-slave-q06p krb5kdc[29091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:43.196978 29171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df: No bootstrap required, opened a new log
I20260929 02:10:43.202967 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:43.182009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.11.182.254:39203)
0929 02:10:43.182894 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:43.182914 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:43.182954 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:10:43.184588 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:43.184607 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:43.184981 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:43.185301 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:43.187076 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:43.188368 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:43.190200 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:43.190938 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:43.191067 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:43.195899 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:43.195907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:43.195921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:43.195953 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:43.199003 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:43.199659 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:43.199671 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:43.199685 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:43.199857 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:43.200252 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:43.200259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:43.200266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:43.200747 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:43.201070 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:43.201543 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 02:10:43.201942 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":394,"threads_started":1}
I20260929 02:10:43.204844 29171 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2763ae3d60c47e6b70cfec559cf55df" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39203 } }
I20260929 02:10:43.205180 29171 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:43.205303 29171 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2763ae3d60c47e6b70cfec559cf55df, State: Initialized, Role: FOLLOWER
I20260929 02:10:43.205875 29171 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2763ae3d60c47e6b70cfec559cf55df" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39203 } }
I20260929 02:10:43.206080 29171 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:10:43.206173 29171 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:10:43.206292 29171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:43.208581 29171 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2763ae3d60c47e6b70cfec559cf55df" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39203 } }
I20260929 02:10:43.209007 29171 leader_election.cc:302] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2763ae3d60c47e6b70cfec559cf55df; no voters:
I20260929 02:10:43.209507 29171 leader_election.cc:288] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:10:43.209648 29176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:43.210345 29176 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [term 1 LEADER]: Becoming Leader. State: Replica: b2763ae3d60c47e6b70cfec559cf55df, State: Running, Role: LEADER
I20260929 02:10:43.210958 29176 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2763ae3d60c47e6b70cfec559cf55df" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39203 } }
I20260929 02:10:43.211519 29171 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:10:43.213754 29177 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2763ae3d60c47e6b70cfec559cf55df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2763ae3d60c47e6b70cfec559cf55df" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39203 } } }
I20260929 02:10:43.214210 29177 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:43.220189 29185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:10:43.220039 29178 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2763ae3d60c47e6b70cfec559cf55df. Latest consensus state: current_term: 1 leader_uuid: "b2763ae3d60c47e6b70cfec559cf55df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2763ae3d60c47e6b70cfec559cf55df" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39203 } } }
I20260929 02:10:43.220455 29178 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:43.225638 29185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:10:43.234302 29185 catalog_manager.cc:1347] Generated new cluster ID: 62d5eeedab79419b9bd59dc85fc2faba
I20260929 02:10:43.234426 29185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:10:43.258946 29185 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:10:43.260872 29185 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:10:43.269869 29185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df: Generated new TSK 0
I20260929 02:10:43.270960 29185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:10:43.339107 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:39203
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:43.675266 29197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:43.675580 29197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:43.675725 29197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:43.682792 29197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:43.682960 29197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:43.683141 29197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:10:43.692683 29197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:39203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:43.693969 29197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:43.695282 29197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:43.708016 29202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:43.708052 29203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:43.707981 29205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:43.708774 29197 server_base.cc:1034] running on GCE node
I20260929 02:10:43.709319 29197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:43.710172 29197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:43.711359 29197 hybrid_clock.cc:648] HybridClock initialized: now 1790647843711323 us; error 27 us; skew 500 ppm
I20260929 02:10:43.711674 29197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:43 dist-test-slave-q06p krb5kdc[29091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647843, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:43.714658 29197 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:10:43.716008 29197 webserver.cc:460] Webserver started at http://127.11.182.193:34347/ using document root <none> and password file <none>
I20260929 02:10:43.716645 29197 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:43.716784 29197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:43.717096 29197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:43.720014 29197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "383fc5f44cd747b7b70d467284aee2c0"
format_stamp: "Formatted at 2026-09-29 02:10:43 on dist-test-slave-q06p"
server_key: "ae6998051bd12619966df875fb2321e4"
server_key_iv: "832ff3521d9b181d74f82a192c611bb1"
server_key_version: "encryptionkey@0"
I20260929 02:10:43.720832 29197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "383fc5f44cd747b7b70d467284aee2c0"
format_stamp: "Formatted at 2026-09-29 02:10:43 on dist-test-slave-q06p"
server_key: "ae6998051bd12619966df875fb2321e4"
server_key_iv: "832ff3521d9b181d74f82a192c611bb1"
server_key_version: "encryptionkey@0"
I20260929 02:10:43.725867 29197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:10:43.729061 29212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:43.730213 29197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 02:10:43.730379 29197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "383fc5f44cd747b7b70d467284aee2c0"
format_stamp: "Formatted at 2026-09-29 02:10:43 on dist-test-slave-q06p"
server_key: "ae6998051bd12619966df875fb2321e4"
server_key_iv: "832ff3521d9b181d74f82a192c611bb1"
server_key_version: "encryptionkey@0"
I20260929 02:10:43.730572 29197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:43.742376 29197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:43.746016 29197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:43.746295 29197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:43.747191 29197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:43.748414 29197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:43.748517 29197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:43.748672 29197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:43.748740 29197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:43.781917 29197 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:42631
I20260929 02:10:43.781956 29324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:42631 every 8 connection(s)
I20260929 02:10:43.783247 29197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:10:43.785250 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 29197
I20260929 02:10:43.785471 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:10:43.786264 11995 external_mini_cluster.cc:1428] Setting key 8443b22f31fb0c33bc47d25fd1090bce
Sep 29 02:10:43 dist-test-slave-q06p krb5kdc[29091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647843, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:43.804847 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:43.785945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:34017 (local address 127.11.182.254:39203)
0929 02:10:43.786467 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:43.786473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:43.787683 (+ 1210us) server_negotiation.cc:407] Connection header received
0929 02:10:43.789754 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:43.789762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:43.789861 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:43.789975 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:43.791664 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:43.792787 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:43.794367 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:43.795127 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:43.795284 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:43.799344 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:43.799348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:43.799352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:43.799382 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:43.801824 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:43.802513 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:43.802522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:43.802525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:43.802637 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:43.803138 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:43.803142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:43.803147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:43.803405 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:43.803562 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:43.804263 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 02:10:43.804486 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":246,"threads_started":1}
I20260929 02:10:43.805774 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:43.786621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:39203 (local address 127.11.182.193:34017)
0929 02:10:43.787501 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:43.787555 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:43.789477 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:43.790224 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:43.790232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:43.790749 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:43.791482 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:43.791501 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:43.792959 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:43.792963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:43.794217 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:43.794230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:43.795351 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:43.795357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:43.795606 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:43.796398 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:43.796450 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:43.799135 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:43.801960 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:43.801968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:43.801983 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:43.802389 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:43.802745 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:43.802750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:43.802753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:43.803019 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:43.803532 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:43.803540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:43.803875 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 02:10:43.804129 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 02:10:43.804658 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":375,"threads_started":1}
I20260929 02:10:43.807531 29325 heartbeater.cc:347] Connected to a master server at 127.11.182.254:39203
I20260929 02:10:43.807848 29325 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:43.808538 29325 heartbeater.cc:511] Master 127.11.182.254:39203 requested a full tablet report, sending...
I20260929 02:10:43.810583 29136 ts_manager.cc:194] Registered new tserver with Master: 383fc5f44cd747b7b70d467284aee2c0 (127.11.182.193:42631)
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:10:43.812207 29136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:34017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:10:43.842120 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:39203
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:44.157915 29331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:44.158174 29331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:44.158337 29331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:44.165439 29331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:44.165545 29331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:44.165753 29331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:10:44.178336 29331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:39203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:44.179773 29331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:44.181382 29331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:44.195533 29337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:44.195598 29336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:44.195605 29339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:44.196877 29331 server_base.cc:1034] running on GCE node
I20260929 02:10:44.197492 29331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:44.198364 29331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:44.199586 29331 hybrid_clock.cc:648] HybridClock initialized: now 1790647844199552 us; error 47 us; skew 500 ppm
I20260929 02:10:44.199900 29331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:44 dist-test-slave-q06p krb5kdc[29091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647844, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:44.202803 29331 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:10:44.204182 29331 webserver.cc:460] Webserver started at http://127.11.182.194:41147/ using document root <none> and password file <none>
I20260929 02:10:44.204828 29331 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:44.204972 29331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:44.205277 29331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:44.208083 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "bae20bd3af274a85b29a2a790ebbf9a3"
format_stamp: "Formatted at 2026-09-29 02:10:44 on dist-test-slave-q06p"
server_key: "207fcfcd1c3c2f7576ff69ceafdefc65"
server_key_iv: "44e6090c93d45b2f3dcba48b9b70b8f9"
server_key_version: "encryptionkey@0"
I20260929 02:10:44.208829 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "bae20bd3af274a85b29a2a790ebbf9a3"
format_stamp: "Formatted at 2026-09-29 02:10:44 on dist-test-slave-q06p"
server_key: "207fcfcd1c3c2f7576ff69ceafdefc65"
server_key_iv: "44e6090c93d45b2f3dcba48b9b70b8f9"
server_key_version: "encryptionkey@0"
I20260929 02:10:44.214041 29331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:10:44.217350 29346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:44.218617 29331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:10:44.218796 29331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "bae20bd3af274a85b29a2a790ebbf9a3"
format_stamp: "Formatted at 2026-09-29 02:10:44 on dist-test-slave-q06p"
server_key: "207fcfcd1c3c2f7576ff69ceafdefc65"
server_key_iv: "44e6090c93d45b2f3dcba48b9b70b8f9"
server_key_version: "encryptionkey@0"
I20260929 02:10:44.219055 29331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:44.241294 29331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:44.245373 29331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:44.245671 29331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:44.246611 29331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:44.247874 29331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:44.247982 29331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:44.248128 29331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:44.248206 29331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:44.280593 29331 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:40235
I20260929 02:10:44.280678 29458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:40235 every 8 connection(s)
I20260929 02:10:44.281850 29331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:10:44.283538 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 29331
I20260929 02:10:44.283740 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:10:44.284197 11995 external_mini_cluster.cc:1428] Setting key 0a55e5e73616055f5cd543e485f4d64f
Sep 29 02:10:44 dist-test-slave-q06p krb5kdc[29091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647844, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:44.302405 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.284432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:59705 (local address 127.11.182.254:39203)
0929 02:10:44.284652 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:44.284658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:44.286079 (+ 1421us) server_negotiation.cc:407] Connection header received
0929 02:10:44.288003 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:44.288011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:44.288097 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:44.288216 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:44.289584 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.290729 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.292260 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.292997 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.293140 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.296888 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:44.296893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:44.296897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:44.296926 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:44.299521 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:44.300210 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:44.300222 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:44.300224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:44.300346 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:44.300769 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:44.300774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:44.300777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:44.301044 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:44.301205 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:44.301834 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 02:10:44.302055 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:10:44.303411 29461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.284926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:39203 (local address 127.11.182.194:59705)
0929 02:10:44.285940 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:44.285991 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:44.287727 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:44.288399 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:44.288406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:44.288898 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:44.289418 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.289434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.290927 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.290931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.292118 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.292130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.293193 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.293197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.293377 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.294107 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:44.294138 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:44.296672 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:44.299665 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:44.299674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:44.299688 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:44.300081 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:44.300455 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:44.300461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:44.300465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:44.300652 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:44.301168 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:44.301176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:44.301491 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 02:10:44.301745 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 02:10:44.302235 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":357,"threads_started":1}
I20260929 02:10:44.304849 29459 heartbeater.cc:347] Connected to a master server at 127.11.182.254:39203
I20260929 02:10:44.305155 29459 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:44.305819 29459 heartbeater.cc:511] Master 127.11.182.254:39203 requested a full tablet report, sending...
I20260929 02:10:44.307219 29136 ts_manager.cc:194] Registered new tserver with Master: bae20bd3af274a85b29a2a790ebbf9a3 (127.11.182.194:40235)
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
I20260929 02:10:44.308073 29136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:59705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:10:44.338364 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:39203
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:44.657214 29464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:44.657476 29464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:44.657661 29464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:44.664721 29464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:44.664819 29464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:44.665001 29464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:10:44.674247 29464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:39203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:44.675611 29464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:44.676832 29464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:44.688872 29470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:44.688992 29469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:44.688892 29472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:44.689873 29464 server_base.cc:1034] running on GCE node
I20260929 02:10:44.690411 29464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:44.691331 29464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:44.692544 29464 hybrid_clock.cc:648] HybridClock initialized: now 1790647844692513 us; error 41 us; skew 500 ppm
I20260929 02:10:44.692853 29464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:44 dist-test-slave-q06p krb5kdc[29091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647844, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:44.695941 29464 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:10:44.697229 29464 webserver.cc:460] Webserver started at http://127.11.182.195:38103/ using document root <none> and password file <none>
I20260929 02:10:44.697835 29464 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:44.697929 29464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:44.698175 29464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:44.700991 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "3d77ba3910c34e9c98b78f4b60daa827"
format_stamp: "Formatted at 2026-09-29 02:10:44 on dist-test-slave-q06p"
server_key: "b6f71adc59d267107ad0f70122337f82"
server_key_iv: "39608d810f06caeec37cfbe51797ecb5"
server_key_version: "encryptionkey@0"
I20260929 02:10:44.701684 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "3d77ba3910c34e9c98b78f4b60daa827"
format_stamp: "Formatted at 2026-09-29 02:10:44 on dist-test-slave-q06p"
server_key: "b6f71adc59d267107ad0f70122337f82"
server_key_iv: "39608d810f06caeec37cfbe51797ecb5"
server_key_version: "encryptionkey@0"
I20260929 02:10:44.706473 29464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 02:10:44.709754 29479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:44.711009 29464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:10:44.711126 29464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "3d77ba3910c34e9c98b78f4b60daa827"
format_stamp: "Formatted at 2026-09-29 02:10:44 on dist-test-slave-q06p"
server_key: "b6f71adc59d267107ad0f70122337f82"
server_key_iv: "39608d810f06caeec37cfbe51797ecb5"
server_key_version: "encryptionkey@0"
I20260929 02:10:44.711270 29464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:44.728632 29464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:44.732338 29464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:44.732553 29464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:44.733351 29464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:44.734712 29464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:44.734781 29464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:44.734933 29464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:44.734982 29464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 02:10:44.767220 29464 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:34833
I20260929 02:10:44.767251 29591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:34833 every 8 connection(s)
I20260929 02:10:44.768515 29464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:10:44.770507 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 29464
I20260929 02:10:44.770711 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:10:44.771387 11995 external_mini_cluster.cc:1428] Setting key 9cdd30f673f84d3a50fadd2b081955a8
Sep 29 02:10:44 dist-test-slave-q06p krb5kdc[29091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647844, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:44.789697 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.771176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:55681 (local address 127.11.182.254:39203)
0929 02:10:44.771414 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:44.771421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:44.772664 (+ 1243us) server_negotiation.cc:407] Connection header received
0929 02:10:44.774783 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:44.774789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:44.774904 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:44.775062 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:44.776573 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.777395 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.778996 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.779406 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.779544 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.782769 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:44.782777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:44.782786 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:44.782818 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:44.786231 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:44.787370 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:44.787374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:44.787377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:44.787508 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:44.788073 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:44.788077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:44.788079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:44.788379 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:44.788496 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:44.789273 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 02:10:44.789436 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 02:10:44.790989 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.771671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:39203 (local address 127.11.182.195:55681)
0929 02:10:44.772487 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:44.772542 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:44.774501 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:44.775327 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:44.775338 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:44.775886 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:44.776410 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.776426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.777568 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.777580 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.778794 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.778805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.779637 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.779645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.779890 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.780460 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:44.780485 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:44.782568 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:44.786467 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:44.786479 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:44.786495 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:44.787221 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:44.787659 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:44.787664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:44.787668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:44.787930 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:44.788519 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:44.788525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:44.788884 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 02:10:44.789157 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 02:10:44.789538 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":333,"threads_started":1}
I20260929 02:10:44.792358 29592 heartbeater.cc:347] Connected to a master server at 127.11.182.254:39203
I20260929 02:10:44.792692 29592 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:44.793465 29592 heartbeater.cc:511] Master 127.11.182.254:39203 requested a full tablet report, sending...
I20260929 02:10:44.795012 29136 ts_manager.cc:194] Registered new tserver with Master: 3d77ba3910c34e9c98b78f4b60daa827 (127.11.182.195:34833)
I20260929 02:10:44.795779 29136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:55681
I20260929 02:10:44.798612 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:10:44.814621 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.802719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.11.182.254:39203)
0929 02:10:44.802939 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:44.802943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:44.803381 (+ 438us) server_negotiation.cc:407] Connection header received
0929 02:10:44.803510 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:44.803513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:44.803566 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:44.803643 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:44.805093 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.805760 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.807036 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.807429 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.807517 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.809770 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:44.809775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:44.809778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:44.809808 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:44.812412 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:44.812984 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:44.812988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:44.812990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:44.813110 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:44.813407 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:44.813410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:44.813412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:44.813687 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:44.813875 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:44.814139 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 02:10:44.814336 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:10:44.815459 29325 heartbeater.cc:503] Master 127.11.182.254:39203 was elected leader, sending a full tablet report...
I20260929 02:10:44.818728 29136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:10:44.825624 29136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:10:44.858192 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.836756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.11.182.193:42631)
0929 02:10:44.850135 (+ 13379us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:44.850143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:44.850171 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:10:44.850269 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:44.850279 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:44.850540 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:44.850815 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:44.851671 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.853053 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.855566 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.857039 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.857255 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.857443 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:44.857693 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 02:10:44.857864 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13142,"thread_start_us":12984,"threads_started":1}
I20260929 02:10:44.858187 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.836591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:42631 (local address 127.0.0.1:58644)
0929 02:10:44.848706 (+ 12115us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:44.848766 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:44.848973 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:44.850989 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:44.850996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:44.851025 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:44.851433 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.851458 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.853213 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.853218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.855411 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.855423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.857234 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.857241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.857429 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.857480 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 02:10:44.857613 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 02:10:44.857806 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11791,"thread_start_us":13092,"threads_started":1}
I20260929 02:10:44.858814 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.837057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.11.182.195:34833)
0929 02:10:44.849346 (+ 12289us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:44.849359 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:44.849655 (+ 296us) server_negotiation.cc:407] Connection header received
0929 02:10:44.850543 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:44.850551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:44.850808 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:44.851005 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:44.852009 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.853761 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.855831 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.857175 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.857354 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.857493 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:44.858276 (+ 783us) server_negotiation.cc:299] Negotiation successful
0929 02:10:44.858479 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12059,"thread_start_us":12286,"threads_started":1}
I20260929 02:10:44.859216 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.836636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:34833 (local address 127.0.0.1:54942)
0929 02:10:44.849547 (+ 12911us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:44.849563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:44.850342 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:44.851324 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:44.851329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:44.851356 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:44.851839 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.851847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.853882 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.853891 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.855659 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.855683 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.857609 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.857615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.857773 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.857818 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 02:10:44.857905 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 02:10:44.858036 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12809,"thread_start_us":12700,"threads_started":1}
I20260929 02:10:44.859761 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.850124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:40235 (local address 127.0.0.1:34108)
0929 02:10:44.851968 (+ 1844us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:44.851984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:44.852099 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:44.852687 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:44.852692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:44.852716 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:44.853043 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.853053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.854915 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.854921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.856894 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:44.856908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.859074 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.859080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:44.859247 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.859277 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:10:44.859360 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 02:10:44.859483 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1704,"thread_start_us":1855,"threads_started":1}
I20260929 02:10:44.860597 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:44.850394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.11.182.194:40235)
0929 02:10:44.851445 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:44.851453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:44.852190 (+ 737us) server_negotiation.cc:407] Connection header received
0929 02:10:44.852302 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:44.852309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:44.852503 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:44.852693 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:44.853216 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.854750 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.857067 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:44.858556 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:44.858733 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:44.858940 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:44.860097 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0929 02:10:44.860286 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":838,"threads_started":1}
I20260929 02:10:44.862600 29260 tablet_service.cc:1467] Processing CreateTablet for tablet ec7e8c06c4164d178ef36bd4663a10b8 (DEFAULT_TABLE table=test-table [id=efe6ab3042ee4dfba1499e53079ea043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:44.862721 29527 tablet_service.cc:1467] Processing CreateTablet for tablet ec7e8c06c4164d178ef36bd4663a10b8 (DEFAULT_TABLE table=test-table [id=efe6ab3042ee4dfba1499e53079ea043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:44.863900 29260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7e8c06c4164d178ef36bd4663a10b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:44.864122 29527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7e8c06c4164d178ef36bd4663a10b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:44.863968 29394 tablet_service.cc:1467] Processing CreateTablet for tablet ec7e8c06c4164d178ef36bd4663a10b8 (DEFAULT_TABLE table=test-table [id=efe6ab3042ee4dfba1499e53079ea043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:44.865233 29394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7e8c06c4164d178ef36bd4663a10b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:44.875172 29610 tablet_bootstrap.cc:492] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: Bootstrap starting.
I20260929 02:10:44.878271 29611 tablet_bootstrap.cc:492] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: Bootstrap starting.
I20260929 02:10:44.878801 29610 tablet_bootstrap.cc:654] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:44.880566 29610 log.cc:824] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:44.881924 29611 tablet_bootstrap.cc:654] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:44.883015 29612 tablet_bootstrap.cc:492] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: Bootstrap starting.
I20260929 02:10:44.883175 29611 log.cc:824] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:44.883903 29610 tablet_bootstrap.cc:492] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: No bootstrap required, opened a new log
I20260929 02:10:44.884200 29610 ts_tablet_manager.cc:1397] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:10:44.886060 29612 tablet_bootstrap.cc:654] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:44.886307 29611 tablet_bootstrap.cc:492] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: No bootstrap required, opened a new log
I20260929 02:10:44.886564 29611 ts_tablet_manager.cc:1397] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 02:10:44.887187 29612 log.cc:824] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:44.889873 29612 tablet_bootstrap.cc:492] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: No bootstrap required, opened a new log
I20260929 02:10:44.890122 29612 ts_tablet_manager.cc:1397] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 02:10:44.894747 29610 raft_consensus.cc:348] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } }
I20260929 02:10:44.895215 29610 raft_consensus.cc:374] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:44.895311 29610 raft_consensus.cc:729] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383fc5f44cd747b7b70d467284aee2c0, State: Initialized, Role: FOLLOWER
I20260929 02:10:44.895947 29610 consensus_queue.cc:260] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } }
I20260929 02:10:44.896917 29611 raft_consensus.cc:348] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } }
I20260929 02:10:44.897403 29611 raft_consensus.cc:374] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:44.897547 29611 raft_consensus.cc:729] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d77ba3910c34e9c98b78f4b60daa827, State: Initialized, Role: FOLLOWER
I20260929 02:10:44.897701 29610 ts_tablet_manager.cc:1428] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 02:10:44.897508 29612 raft_consensus.cc:348] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } }
I20260929 02:10:44.897925 29612 raft_consensus.cc:374] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:44.898054 29612 raft_consensus.cc:729] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae20bd3af274a85b29a2a790ebbf9a3, State: Initialized, Role: FOLLOWER
I20260929 02:10:44.898327 29611 consensus_queue.cc:260] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } }
I20260929 02:10:44.898619 29612 consensus_queue.cc:260] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } }
I20260929 02:10:44.900822 29592 heartbeater.cc:503] Master 127.11.182.254:39203 was elected leader, sending a full tablet report...
I20260929 02:10:44.901592 29611 ts_tablet_manager.cc:1428] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260929 02:10:44.902878 29459 heartbeater.cc:503] Master 127.11.182.254:39203 was elected leader, sending a full tablet report...
I20260929 02:10:44.903508 29612 ts_tablet_manager.cc:1428] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260929 02:10:45.020210 29593 tablet.cc:2369] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:10:45.033907 29460 tablet.cc:2369] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:10:45.035705 29326 tablet.cc:2369] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:45.061605 29618 raft_consensus.cc:482] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:10:45.061885 29618 raft_consensus.cc:504] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } }
I20260929 02:10:45.063633 29618 leader_election.cc:288] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 383fc5f44cd747b7b70d467284aee2c0 (127.11.182.193:42631), 3d77ba3910c34e9c98b78f4b60daa827 (127.11.182.195:34833)
I20260929 02:10:45.070276 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:45.064280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:35429 (local address 127.11.182.195:34833)
0929 02:10:45.064487 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:45.064496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:45.065056 (+ 560us) server_negotiation.cc:407] Connection header received
0929 02:10:45.065242 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:45.065246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:45.065331 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:45.065423 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:45.066052 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.066781 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.068513 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.069287 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.069408 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:45.069498 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:45.069878 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 02:10:45.069999 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:10:45.070286 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:45.064011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:38343 (local address 127.11.182.193:42631)
0929 02:10:45.064251 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:45.064260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:45.064947 (+ 687us) server_negotiation.cc:407] Connection header received
0929 02:10:45.065145 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:45.065154 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:45.065243 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:45.065320 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:45.065959 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.066695 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.068587 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.069379 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.069538 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:45.069612 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:45.069921 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 02:10:45.070034 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:10:45.070453 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:45.064069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:34833 (local address 127.11.182.194:35429)
0929 02:10:45.064934 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:45.064953 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:45.065104 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:45.065470 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:45.065474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:45.065514 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:45.065906 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:45.065917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.066956 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.066960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:45.068369 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:45.068380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.069512 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.069520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:45.069712 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:45.069738 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 02:10:45.069824 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:10:45.069957 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":2,"thread_start_us":418,"threads_started":1}
I20260929 02:10:45.070509 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:45.063926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:42631 (local address 127.11.182.194:38343)
0929 02:10:45.064836 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:45.064852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:45.064980 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:45.065377 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:45.065383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:45.065411 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:45.065812 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:45.065824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.066824 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.066832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:45.068411 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:45.068422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.069604 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.069609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:45.069736 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:45.069757 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 02:10:45.069877 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 02:10:45.069995 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":47,"thread_start_us":664,"threads_started":1}
I20260929 02:10:45.071550 29547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7e8c06c4164d178ef36bd4663a10b8" candidate_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d77ba3910c34e9c98b78f4b60daa827" is_pre_election: true
I20260929 02:10:45.071554 29280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7e8c06c4164d178ef36bd4663a10b8" candidate_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383fc5f44cd747b7b70d467284aee2c0" is_pre_election: true
I20260929 02:10:45.072028 29280 raft_consensus.cc:2454] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae20bd3af274a85b29a2a790ebbf9a3 in term 0.
I20260929 02:10:45.072036 29547 raft_consensus.cc:2454] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae20bd3af274a85b29a2a790ebbf9a3 in term 0.
I20260929 02:10:45.072695 29348 leader_election.cc:302] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383fc5f44cd747b7b70d467284aee2c0, bae20bd3af274a85b29a2a790ebbf9a3; no voters:
I20260929 02:10:45.073082 29618 raft_consensus.cc:2781] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:10:45.073243 29618 raft_consensus.cc:482] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:10:45.073347 29618 raft_consensus.cc:3037] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:45.075546 29618 raft_consensus.cc:504] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } }
I20260929 02:10:45.076213 29618 leader_election.cc:288] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [CANDIDATE]: Term 1 election: Requested vote from peers 383fc5f44cd747b7b70d467284aee2c0 (127.11.182.193:42631), 3d77ba3910c34e9c98b78f4b60daa827 (127.11.182.195:34833)
I20260929 02:10:45.076596 29280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7e8c06c4164d178ef36bd4663a10b8" candidate_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383fc5f44cd747b7b70d467284aee2c0"
I20260929 02:10:45.076728 29547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7e8c06c4164d178ef36bd4663a10b8" candidate_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d77ba3910c34e9c98b78f4b60daa827"
I20260929 02:10:45.076838 29280 raft_consensus.cc:3037] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:45.076920 29547 raft_consensus.cc:3037] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:45.080129 29547 raft_consensus.cc:2454] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae20bd3af274a85b29a2a790ebbf9a3 in term 1.
I20260929 02:10:45.080139 29280 raft_consensus.cc:2454] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae20bd3af274a85b29a2a790ebbf9a3 in term 1.
I20260929 02:10:45.080554 29348 leader_election.cc:302] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 383fc5f44cd747b7b70d467284aee2c0, 3d77ba3910c34e9c98b78f4b60daa827, bae20bd3af274a85b29a2a790ebbf9a3; no voters:
I20260929 02:10:45.080891 29618 raft_consensus.cc:2781] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:45.081316 29618 raft_consensus.cc:686] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 1 LEADER]: Becoming Leader. State: Replica: bae20bd3af274a85b29a2a790ebbf9a3, State: Running, Role: LEADER
I20260929 02:10:45.081864 29618 consensus_queue.cc:237] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } }
I20260929 02:10:45.086665 29136 catalog_manager.cc:5495] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bae20bd3af274a85b29a2a790ebbf9a3 (127.11.182.194). New cstate: current_term: 1 leader_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae20bd3af274a85b29a2a790ebbf9a3" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 40235 } health_report { overall_health: HEALTHY } } }
I20260929 02:10:45.154208 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:45.149195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.11.182.194:40235)
0929 02:10:45.149419 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:45.149423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:45.149440 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:10:45.149524 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:45.149528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:45.149609 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:45.149730 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:10:45.150053 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.150776 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.152196 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.152596 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.152687 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:45.153117 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:10:45.153610 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:10:45.153714 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:45.153826 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 02:10:45.153888 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 02:10:45.162073 29547 raft_consensus.cc:1261] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [term 1 FOLLOWER]: Refusing update from remote peer bae20bd3af274a85b29a2a790ebbf9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:45.162108 29280 raft_consensus.cc:1261] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [term 1 FOLLOWER]: Refusing update from remote peer bae20bd3af274a85b29a2a790ebbf9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:45.163127 29621 consensus_queue.cc:1035] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d77ba3910c34e9c98b78f4b60daa827" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:45.163460 29618 consensus_queue.cc:1035] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "383fc5f44cd747b7b70d467284aee2c0" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:45.174091 29624 mvcc.cc:205] Tried to move back new op lower bound from 7334493573774614528 to 7334493573458767872. Current Snapshot: MvccSnapshot[applied={T|T < 7334493573774614528}]
I20260929 02:10:45.200562 29135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:45.200906 29135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:45.204496 29135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2763ae3d60c47e6b70cfec559cf55df: Sending DeleteTablet for 3 replicas of tablet ec7e8c06c4164d178ef36bd4663a10b8
I20260929 02:10:45.205593 29527 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7e8c06c4164d178ef36bd4663a10b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:45 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:54942
I20260929 02:10:45.205705 29394 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7e8c06c4164d178ef36bd4663a10b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:45 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:34108
I20260929 02:10:45.205691 29260 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7e8c06c4164d178ef36bd4663a10b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:45 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:58644
I20260929 02:10:45.208690 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 29197
I20260929 02:10:45.222591 29633 tablet_replica.cc:330] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: stopping tablet replica
I20260929 02:10:45.223117 29633 raft_consensus.cc:2229] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:45.223686 29633 raft_consensus.cc:2258] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:45.224489 29635 tablet_replica.cc:330] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: stopping tablet replica
I20260929 02:10:45.224934 29635 raft_consensus.cc:2229] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:45.225440 29635 raft_consensus.cc:2258] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:45.229090 29635 ts_tablet_manager.cc:1905] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:45.234203 29634 tablet_replica.cc:330] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: stopping tablet replica
Sep 29 02:10:45 dist-test-slave-q06p krb5kdc[29091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:10:45.234650 29634 raft_consensus.cc:2229] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:10:45.235631 29634 raft_consensus.cc:2258] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:45.236486 29633 ts_tablet_manager.cc:1905] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:45.238286 29635 ts_tablet_manager.cc:1918] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:45.238474 29635 log.cc:1196] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/ec7e8c06c4164d178ef36bd4663a10b8
I20260929 02:10:45.239091 29635 ts_tablet_manager.cc:1939] T ec7e8c06c4164d178ef36bd4663a10b8 P 383fc5f44cd747b7b70d467284aee2c0: Deleting consensus metadata
I20260929 02:10:45.241031 29121 catalog_manager.cc:4841] TS 383fc5f44cd747b7b70d467284aee2c0 (127.11.182.193:42631): tablet ec7e8c06c4164d178ef36bd4663a10b8 (table test-table [id=efe6ab3042ee4dfba1499e53079ea043]) successfully deleted
I20260929 02:10:45.243271 29634 ts_tablet_manager.cc:1905] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:45.243383 29633 ts_tablet_manager.cc:1918] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:45.243530 29633 log.cc:1196] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/ec7e8c06c4164d178ef36bd4663a10b8
I20260929 02:10:45.244046 29633 ts_tablet_manager.cc:1939] T ec7e8c06c4164d178ef36bd4663a10b8 P 3d77ba3910c34e9c98b78f4b60daa827: Deleting consensus metadata
I20260929 02:10:45.244449 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:45.210497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.11.182.193:42631)
0929 02:10:45.210746 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:45.210755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:45.214352 (+ 3597us) server_negotiation.cc:407] Connection header received
0929 02:10:45.215288 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:45.215295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:45.215380 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:45.215468 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:45.218947 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.220135 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.225958 (+ 5823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.226621 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.226733 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:45.236084 (+ 9351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:45.236099 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:45.236116 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:45.236186 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:45.240350 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:45.241824 (+ 1474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:45.241833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:45.241837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:45.241996 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:45.242466 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:45.242472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:45.242476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:45.242949 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:45.243236 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:45.243832 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 02:10:45.244063 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:10:45.245771 29122 catalog_manager.cc:4841] TS 3d77ba3910c34e9c98b78f4b60daa827 (127.11.182.195:34833): tablet ec7e8c06c4164d178ef36bd4663a10b8 (table test-table [id=efe6ab3042ee4dfba1499e53079ea043]) successfully deleted
I20260929 02:10:45.245918 29320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:45.249123 29634 ts_tablet_manager.cc:1918] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:45.249323 29634 log.cc:1196] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/ec7e8c06c4164d178ef36bd4663a10b8
I20260929 02:10:45.249917 29634 ts_tablet_manager.cc:1939] T ec7e8c06c4164d178ef36bd4663a10b8 P bae20bd3af274a85b29a2a790ebbf9a3: Deleting consensus metadata
I20260929 02:10:45.251471 29121 catalog_manager.cc:4841] TS bae20bd3af274a85b29a2a790ebbf9a3 (127.11.182.194:40235): tablet ec7e8c06c4164d178ef36bd4663a10b8 (table test-table [id=efe6ab3042ee4dfba1499e53079ea043]) successfully deleted
I20260929 02:10:45.339977 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 29197
I20260929 02:10:45.358227 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 29331
Sep 29 02:10:45 dist-test-slave-q06p krb5kdc[29091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:10:45.377640 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:45.359282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42134 (local address 127.11.182.194:40235)
0929 02:10:45.359454 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:45.359459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:45.359476 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:10:45.359530 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:45.359534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:45.359594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:45.359678 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:45.361346 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.362438 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.364414 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.364895 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.364998 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:45.370142 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:45.370164 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:45.370185 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:45.370235 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:45.374337 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:45.375229 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:45.375239 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:45.375243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:45.375430 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:45.375824 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:45.375829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:45.375833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:45.376307 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:45.376709 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:45.377002 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 02:10:45.377227 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:10:45.378474 29454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:45.492929 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 29331
I20260929 02:10:45.512290 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 29464
Sep 29 02:10:45 dist-test-slave-q06p krb5kdc[29091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:10:45.527770 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:45.513000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.11.182.195:34833)
0929 02:10:45.513181 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:45.513188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:45.513259 (+ 71us) server_negotiation.cc:407] Connection header received
0929 02:10:45.513495 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:45.513504 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:45.513572 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:45.513671 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:45.514955 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.515651 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.516838 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:45.517213 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:45.517367 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:45.521994 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:45.522009 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:45.522025 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:45.522058 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:45.524933 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:45.525676 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:45.525685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:45.525687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:45.525842 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:45.526219 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:45.526225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:45.526228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:45.526571 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:45.526902 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:45.527206 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 02:10:45.527433 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:10:45.528517 29587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:45.617983 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 29464
I20260929 02:10:45.634640 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 29104
I20260929 02:10:45.635819 29166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:45.745580 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 29104
2026-09-29T02:10:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4352 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T02:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:10:45Z Disabled control of system clock
I20260929 02:10:45.822199 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45251
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:37435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:10:46.147784 29644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:46.148088 29644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:46.148170 29644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:46.155735 29644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:10:46.155845 29644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:46.155882 29644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:46.155930 29644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:10:46.155977 29644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:10:46.165985 29644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:45251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:46.167356 29644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:46.169046 29644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:46.180887 29652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:46.180984 29649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:46.180960 29650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:46.181766 29644 server_base.cc:1034] running on GCE node
I20260929 02:10:46.182569 29644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:46.184078 29644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:46.185309 29644 hybrid_clock.cc:648] HybridClock initialized: now 1790647846185266 us; error 65 us; skew 500 ppm
I20260929 02:10:46.185657 29644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:10:46.191782 29644 webserver.cc:460] Webserver started at http://127.11.182.254:36857/ using document root <none> and password file <none>
I20260929 02:10:46.192495 29644 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:46.192631 29644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:46.193030 29644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:46.196034 29644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "2bf62cc5d0a4455d86220f7fefc29370"
format_stamp: "Formatted at 2026-09-29 02:10:46 on dist-test-slave-q06p"
server_key: "ec1f04368490830c5832824a461eda5d"
server_key_iv: "bd016907b72f126bd6486fd67eae2f41"
server_key_version: "encryptionkey@0"
I20260929 02:10:46.196848 29644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "2bf62cc5d0a4455d86220f7fefc29370"
format_stamp: "Formatted at 2026-09-29 02:10:46 on dist-test-slave-q06p"
server_key: "ec1f04368490830c5832824a461eda5d"
server_key_iv: "bd016907b72f126bd6486fd67eae2f41"
server_key_version: "encryptionkey@0"
I20260929 02:10:46.202028 29644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:10:46.205981 29658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:46.207405 29644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:10:46.207599 29644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "2bf62cc5d0a4455d86220f7fefc29370"
format_stamp: "Formatted at 2026-09-29 02:10:46 on dist-test-slave-q06p"
server_key: "ec1f04368490830c5832824a461eda5d"
server_key_iv: "bd016907b72f126bd6486fd67eae2f41"
server_key_version: "encryptionkey@0"
I20260929 02:10:46.207835 29644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:46.218717 29644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:46.219534 29644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:46.219820 29644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:46.238960 29644 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:45251
I20260929 02:10:46.238929 29709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:45251 every 8 connection(s)
I20260929 02:10:46.240305 29644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:10:46.244113 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 29644
I20260929 02:10:46.244130 29710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:46.244318 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:10:46.244753 11995 external_mini_cluster.cc:1428] Setting key c6352e1caebaa9267218a8606c34f077
I20260929 02:10:46.257917 29710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370: Bootstrap starting.
I20260929 02:10:46.259459 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:46.246903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.11.182.254:45251)
0929 02:10:46.248039 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:46.248049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:46.248092 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:10:46.250086 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:46.250110 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:46.250584 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:46.250996 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:46.251612 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:46.252869 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:46.254991 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:46.255659 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:46.255797 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:46.257036 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:46.257043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:46.257057 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:46.257613 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:46.257694 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:46.257715 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:46.257816 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:46.258062 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 02:10:46.258431 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":450,"threads_started":1}
I20260929 02:10:46.261257 29710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:46.262310 29710 log.cc:824] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:46.265079 29710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370: No bootstrap required, opened a new log
I20260929 02:10:46.272276 29710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf62cc5d0a4455d86220f7fefc29370" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45251 } }
I20260929 02:10:46.272655 29710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:46.272783 29710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf62cc5d0a4455d86220f7fefc29370, State: Initialized, Role: FOLLOWER
I20260929 02:10:46.273399 29710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf62cc5d0a4455d86220f7fefc29370" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45251 } }
I20260929 02:10:46.273636 29710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:10:46.273782 29710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:10:46.273936 29710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:46.276224 29710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf62cc5d0a4455d86220f7fefc29370" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45251 } }
I20260929 02:10:46.276721 29710 leader_election.cc:302] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bf62cc5d0a4455d86220f7fefc29370; no voters:
I20260929 02:10:46.277333 29710 leader_election.cc:288] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:10:46.277500 29715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:46.278180 29715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf62cc5d0a4455d86220f7fefc29370, State: Running, Role: LEADER
I20260929 02:10:46.278769 29715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf62cc5d0a4455d86220f7fefc29370" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45251 } }
I20260929 02:10:46.279670 29710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:10:46.281522 29716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bf62cc5d0a4455d86220f7fefc29370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf62cc5d0a4455d86220f7fefc29370" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45251 } } }
I20260929 02:10:46.281847 29716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:46.282187 29717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bf62cc5d0a4455d86220f7fefc29370. Latest consensus state: current_term: 1 leader_uuid: "2bf62cc5d0a4455d86220f7fefc29370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf62cc5d0a4455d86220f7fefc29370" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45251 } } }
I20260929 02:10:46.282521 29717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370 [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:46.284179 29721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:10:46.289146 29721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:10:46.297448 29721 catalog_manager.cc:1347] Generated new cluster ID: 1a065ae6e6b94be4929954bb840dd338
I20260929 02:10:46.297561 29721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:10:46.312752 29721 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:10:46.314033 29721 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:10:46.324213 29721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370: Generated new TSK 0
I20260929 02:10:46.325381 29721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:10:46.333808 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45251
--builtin_ntp_servers=127.11.182.212:37435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:10:46.658968 29734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:46.659242 29734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:46.659335 29734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:46.666389 29734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:46.666477 29734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:46.666576 29734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:10:46.675343 29734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:46.676477 29734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:46.678098 29734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:46.690958 29742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:46.690945 29740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:46.691237 29739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:46.692131 29734 server_base.cc:1034] running on GCE node
I20260929 02:10:46.692700 29734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:46.693616 29734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:46.694883 29734 hybrid_clock.cc:648] HybridClock initialized: now 1790647846694819 us; error 54 us; skew 500 ppm
I20260929 02:10:46.695197 29734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:10:46.697372 29734 webserver.cc:460] Webserver started at http://127.11.182.193:42121/ using document root <none> and password file <none>
I20260929 02:10:46.698035 29734 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:46.698161 29734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:46.698479 29734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:46.701251 29734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "b7a430f6603340c794c51122dc0962de"
format_stamp: "Formatted at 2026-09-29 02:10:46 on dist-test-slave-q06p"
server_key: "8e66d2492799da8027331af1f7cdcd92"
server_key_iv: "47f4d7aa91f6cc4a6ae078cf5d62220e"
server_key_version: "encryptionkey@0"
I20260929 02:10:46.702020 29734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "b7a430f6603340c794c51122dc0962de"
format_stamp: "Formatted at 2026-09-29 02:10:46 on dist-test-slave-q06p"
server_key: "8e66d2492799da8027331af1f7cdcd92"
server_key_iv: "47f4d7aa91f6cc4a6ae078cf5d62220e"
server_key_version: "encryptionkey@0"
I20260929 02:10:46.707070 29734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:10:46.710307 29748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:46.711545 29734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:10:46.711742 29734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "b7a430f6603340c794c51122dc0962de"
format_stamp: "Formatted at 2026-09-29 02:10:46 on dist-test-slave-q06p"
server_key: "8e66d2492799da8027331af1f7cdcd92"
server_key_iv: "47f4d7aa91f6cc4a6ae078cf5d62220e"
server_key_version: "encryptionkey@0"
I20260929 02:10:46.711956 29734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:46.766719 29734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:46.767596 29734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:46.767911 29734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:46.768919 29734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:46.770383 29734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:46.770507 29734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:46.770625 29734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:46.770706 29734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:46.805639 29734 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:38219
I20260929 02:10:46.805724 29860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:38219 every 8 connection(s)
I20260929 02:10:46.806978 29734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:10:46.807477 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 29734
I20260929 02:10:46.807770 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:10:46.808359 11995 external_mini_cluster.cc:1428] Setting key a44cf8630db3f0aa0d1930dbdde7e7b8
I20260929 02:10:46.811344 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45251
--builtin_ntp_servers=127.11.182.212:37435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:10:46.821388 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:46.809608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:46053 (local address 127.11.182.254:45251)
0929 02:10:46.810063 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:46.810068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:46.813431 (+ 3363us) server_negotiation.cc:407] Connection header received
0929 02:10:46.813485 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:46.813489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:46.813549 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:46.813628 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:46.815081 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:46.815906 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:46.818349 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:46.818727 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:46.818822 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:46.820330 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:46.820339 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:46.820347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:46.820374 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:46.820426 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:46.820444 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:46.820539 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:46.820830 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 02:10:46.820944 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":242,"threads_started":1}
I20260929 02:10:46.822248 29864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:46.810353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45251 (local address 127.11.182.193:46053)
0929 02:10:46.811462 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:46.811505 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:46.813165 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:46.813909 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:46.813922 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:46.814055 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:46.814882 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:46.814910 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:46.816048 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:46.816056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:46.818196 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:46.818215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:46.818972 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:46.818982 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:46.819286 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:46.820010 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:10:46.820013 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:46.820051 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:10:46.820132 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:46.820571 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:46.820581 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:46.820588 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:10:46.820766 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 02:10:46.821092 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":336,"threads_started":1}
I20260929 02:10:46.823909 29861 heartbeater.cc:347] Connected to a master server at 127.11.182.254:45251
I20260929 02:10:46.824223 29861 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:46.824960 29861 heartbeater.cc:511] Master 127.11.182.254:45251 requested a full tablet report, sending...
I20260929 02:10:46.827050 29675 ts_manager.cc:194] Registered new tserver with Master: b7a430f6603340c794c51122dc0962de (127.11.182.193:38219)
I20260929 02:10:46.828466 29675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.193:46053
W20260929 02:10:47.133203 29865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:47.133473 29865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:47.133561 29865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:47.140775 29865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:47.140882 29865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:47.140990 29865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:10:47.149827 29865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:47.151013 29865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:47.152679 29865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:47.165220 29873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:47.165323 29871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:47.165835 29870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:47.165923 29865 server_base.cc:1034] running on GCE node
I20260929 02:10:47.166599 29865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:47.167573 29865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:47.168799 29865 hybrid_clock.cc:648] HybridClock initialized: now 1790647847168754 us; error 57 us; skew 500 ppm
I20260929 02:10:47.169101 29865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:10:47.171342 29865 webserver.cc:460] Webserver started at http://127.11.182.194:35047/ using document root <none> and password file <none>
I20260929 02:10:47.171998 29865 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:47.172129 29865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:47.172434 29865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:47.175208 29865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "b92589f236c243d0bcc2cab354f68fd7"
format_stamp: "Formatted at 2026-09-29 02:10:47 on dist-test-slave-q06p"
server_key: "efe1ea2ba3264415b61e995c889312f3"
server_key_iv: "0a43549f05ee924e634064f9361e64d9"
server_key_version: "encryptionkey@0"
I20260929 02:10:47.175967 29865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "b92589f236c243d0bcc2cab354f68fd7"
format_stamp: "Formatted at 2026-09-29 02:10:47 on dist-test-slave-q06p"
server_key: "efe1ea2ba3264415b61e995c889312f3"
server_key_iv: "0a43549f05ee924e634064f9361e64d9"
server_key_version: "encryptionkey@0"
I20260929 02:10:47.180994 29865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:10:47.184394 29879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:47.185781 29865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:10:47.185948 29865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "b92589f236c243d0bcc2cab354f68fd7"
format_stamp: "Formatted at 2026-09-29 02:10:47 on dist-test-slave-q06p"
server_key: "efe1ea2ba3264415b61e995c889312f3"
server_key_iv: "0a43549f05ee924e634064f9361e64d9"
server_key_version: "encryptionkey@0"
I20260929 02:10:47.186151 29865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:47.227661 29865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:47.228524 29865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:47.228839 29865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:47.229892 29865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:47.231513 29865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:47.231633 29865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:47.231768 29865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:47.231863 29865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:47.265493 29865 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:36269
I20260929 02:10:47.265549 29991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:36269 every 8 connection(s)
I20260929 02:10:47.266769 29865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:10:47.271376 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 29865
I20260929 02:10:47.271556 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:10:47.272035 11995 external_mini_cluster.cc:1428] Setting key c5cbc001890c6e3f9c34b376a2b938d9
I20260929 02:10:47.274662 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45251
--builtin_ntp_servers=127.11.182.212:37435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:10:47.279841 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.270020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:54725 (local address 127.11.182.254:45251)
0929 02:10:47.270233 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.270239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.271717 (+ 1478us) server_negotiation.cc:407] Connection header received
0929 02:10:47.273473 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.273478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.273568 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.273674 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:47.274701 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.275559 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.277081 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.277468 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.277601 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.279037 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:47.279042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:47.279045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:47.279063 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:47.279094 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:47.279106 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:47.279204 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.279425 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.279525 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:10:47.280968 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.270525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45251 (local address 127.11.182.194:54725)
0929 02:10:47.271555 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:47.271596 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:47.273212 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:47.273900 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:47.273908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:47.273994 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:47.274540 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.274555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.275723 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.275728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.276932 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.276943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.277683 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.277691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.277921 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.278652 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:10:47.278654 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:47.278691 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:10:47.278772 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:47.279204 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:47.279217 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:47.279227 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:10:47.279412 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 02:10:47.279800 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":359,"threads_started":1}
I20260929 02:10:47.282500 29992 heartbeater.cc:347] Connected to a master server at 127.11.182.254:45251
I20260929 02:10:47.282831 29992 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:47.283751 29992 heartbeater.cc:511] Master 127.11.182.254:45251 requested a full tablet report, sending...
I20260929 02:10:47.285377 29675 ts_manager.cc:194] Registered new tserver with Master: b92589f236c243d0bcc2cab354f68fd7 (127.11.182.194:36269)
I20260929 02:10:47.286334 29675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.194:54725
W20260929 02:10:47.600389 29995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:47.600872 29995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:47.601073 29995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:47.610127 29995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:47.610211 29995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:47.610311 29995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:10:47.619555 29995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:45251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:47.620699 29995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:47.622663 29995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:47.635247 30000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:47.635244 30001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:47.635423 30003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:47.636425 29995 server_base.cc:1034] running on GCE node
I20260929 02:10:47.636978 29995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:47.637773 29995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:47.638942 29995 hybrid_clock.cc:648] HybridClock initialized: now 1790647847638891 us; error 45 us; skew 500 ppm
I20260929 02:10:47.639271 29995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:10:47.641398 29995 webserver.cc:460] Webserver started at http://127.11.182.195:42541/ using document root <none> and password file <none>
I20260929 02:10:47.641999 29995 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:47.642081 29995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:47.642338 29995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:47.645010 29995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "f1006679a7cd4acda70ec44d206139c3"
format_stamp: "Formatted at 2026-09-29 02:10:47 on dist-test-slave-q06p"
server_key: "9527aa607cb00063a6fa460197321270"
server_key_iv: "e4b4c804b9ec5078ef1cb6d93650173b"
server_key_version: "encryptionkey@0"
I20260929 02:10:47.645728 29995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "f1006679a7cd4acda70ec44d206139c3"
format_stamp: "Formatted at 2026-09-29 02:10:47 on dist-test-slave-q06p"
server_key: "9527aa607cb00063a6fa460197321270"
server_key_iv: "e4b4c804b9ec5078ef1cb6d93650173b"
server_key_version: "encryptionkey@0"
I20260929 02:10:47.650447 29995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:10:47.653642 30009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:47.654902 29995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:10:47.655083 29995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "f1006679a7cd4acda70ec44d206139c3"
format_stamp: "Formatted at 2026-09-29 02:10:47 on dist-test-slave-q06p"
server_key: "9527aa607cb00063a6fa460197321270"
server_key_iv: "e4b4c804b9ec5078ef1cb6d93650173b"
server_key_version: "encryptionkey@0"
I20260929 02:10:47.655297 29995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:47.672415 29995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:47.673199 29995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:47.673499 29995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:47.674412 29995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:47.675894 29995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:47.676017 29995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:47.676152 29995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:47.676242 29995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:47.709024 29995 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:45669
I20260929 02:10:47.709074 30121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:45669 every 8 connection(s)
I20260929 02:10:47.710330 29995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:10:47.720508 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 29995
I20260929 02:10:47.720696 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:10:47.721247 11995 external_mini_cluster.cc:1428] Setting key bf0d804a569a2a498cd06c2bbd18385a
I20260929 02:10:47.722872 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.713033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:55683 (local address 127.11.182.254:45251)
0929 02:10:47.713211 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.713216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.714572 (+ 1356us) server_negotiation.cc:407] Connection header received
0929 02:10:47.716473 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.716478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.716553 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.716662 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:47.717682 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.718488 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.720012 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.720418 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.720522 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.722007 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:47.722012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:47.722015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:47.722032 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:47.722062 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:47.722073 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:47.722175 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.722393 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.722504 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:10:47.724084 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.713554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45251 (local address 127.11.182.195:55683)
0929 02:10:47.714411 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:47.714457 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:47.716199 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:47.716850 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:47.716858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:47.716939 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:47.717486 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.717501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.718626 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.718629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.719862 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.719873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.720646 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.720656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.720944 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.721666 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:10:47.721669 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:47.721738 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:10:47.721810 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:47.722173 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:47.722184 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:47.722194 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:10:47.722380 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 02:10:47.722767 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":267,"threads_started":1}
I20260929 02:10:47.725611 30122 heartbeater.cc:347] Connected to a master server at 127.11.182.254:45251
I20260929 02:10:47.725900 30122 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:47.726608 30122 heartbeater.cc:511] Master 127.11.182.254:45251 requested a full tablet report, sending...
I20260929 02:10:47.727929 29675 ts_manager.cc:194] Registered new tserver with Master: f1006679a7cd4acda70ec44d206139c3 (127.11.182.195:45669)
I20260929 02:10:47.728681 29675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.195:55683
I20260929 02:10:47.736368 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:10:47.746053 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.740104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38058 (local address 127.11.182.254:45251)
0929 02:10:47.740327 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.740336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.740741 (+ 405us) server_negotiation.cc:407] Connection header received
0929 02:10:47.740958 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.740968 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.741040 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.741153 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:47.741633 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.742318 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.743714 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.744118 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.744270 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.745365 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:47.745369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:47.745373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:47.745389 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:47.745420 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:47.745431 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:47.745524 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.745677 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.745766 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:10:47.749483 29675 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:10:47.756172 29675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:10:47.779177 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.769824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:36269 (local address 127.0.0.1:52340)
0929 02:10:47.771230 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:47.771251 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:47.771356 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:47.772306 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:47.772311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:47.772332 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:47.772575 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.772582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.774515 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.774520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.776850 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.776861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.778151 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.778162 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.778403 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.778441 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:10:47.778546 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:10:47.778769 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1279,"mutex_wait_us":1,"thread_start_us":725,"threads_started":1}
I20260929 02:10:47.780601 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.770382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.11.182.194:36269)
0929 02:10:47.770897 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.770909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.771483 (+ 574us) server_negotiation.cc:407] Connection header received
0929 02:10:47.771592 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.771600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.771873 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.772058 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:47.772743 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.774344 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.777021 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.777903 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.779773 (+ 1870us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.780021 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.780083 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.780241 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":278,"threads_started":1}
I20260929 02:10:47.780782 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.768220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45669 (local address 127.0.0.1:57476)
0929 02:10:47.770308 (+ 2088us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:47.770326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:47.770428 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:47.773742 (+ 3314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:47.773746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:47.773786 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:47.774209 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.774223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.775909 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.775917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.778130 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.778147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.780047 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.780053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.780219 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.780247 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:10:47.780322 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 02:10:47.780447 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1952,"mutex_wait_us":573,"thread_start_us":275,"threads_started":1}
I20260929 02:10:47.780895 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.769019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57476 (local address 127.11.182.195:45669)
0929 02:10:47.773215 (+ 4196us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.773223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.773253 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:10:47.773347 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.773354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.773546 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.773725 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:47.774413 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.775776 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.778393 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.779840 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.780051 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.780295 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.780365 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.780524 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4024,"thread_start_us":3866,"threads_started":1}
I20260929 02:10:47.781430 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.768212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38219 (local address 127.0.0.1:34430)
0929 02:10:47.769424 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:47.769465 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:47.769619 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:47.774061 (+ 4442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:47.774066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:47.774093 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:47.774528 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.774549 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.776508 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.776513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.778612 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.778626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.780631 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.780638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.780842 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.780884 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:10:47.780952 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 02:10:47.781106 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":1,"thread_start_us":2723,"threads_started":1}
I20260929 02:10:47.783057 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.768295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.11.182.193:38219)
0929 02:10:47.773496 (+ 5201us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.773514 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.773535 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:10:47.773607 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.773611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.773852 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.774060 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:47.774790 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.776343 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.778868 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.780399 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.782346 (+ 1947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.782505 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.782553 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.782710 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5060,"thread_start_us":4974,"threads_started":1}
I20260929 02:10:47.784687 29927 tablet_service.cc:1467] Processing CreateTablet for tablet eff79e1fb41045f2a197f4751c5fd429 (DEFAULT_TABLE table=test-table [id=e28d89a13ff649f48a61b8f850d9f50e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:47.785915 29927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff79e1fb41045f2a197f4751c5fd429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:47.785425 30057 tablet_service.cc:1467] Processing CreateTablet for tablet eff79e1fb41045f2a197f4751c5fd429 (DEFAULT_TABLE table=test-table [id=e28d89a13ff649f48a61b8f850d9f50e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:47.786875 30057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff79e1fb41045f2a197f4751c5fd429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:47.786386 29796 tablet_service.cc:1467] Processing CreateTablet for tablet eff79e1fb41045f2a197f4751c5fd429 (DEFAULT_TABLE table=test-table [id=e28d89a13ff649f48a61b8f850d9f50e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:47.787700 29796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff79e1fb41045f2a197f4751c5fd429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:47.798480 30140 tablet_bootstrap.cc:492] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: Bootstrap starting.
I20260929 02:10:47.801107 30141 tablet_bootstrap.cc:492] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: Bootstrap starting.
I20260929 02:10:47.801995 30140 tablet_bootstrap.cc:654] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:47.802699 30142 tablet_bootstrap.cc:492] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de: Bootstrap starting.
I20260929 02:10:47.803361 30140 log.cc:824] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:47.804602 30141 tablet_bootstrap.cc:654] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:47.805903 30141 log.cc:824] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:47.806200 30142 tablet_bootstrap.cc:654] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:47.806550 30140 tablet_bootstrap.cc:492] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: No bootstrap required, opened a new log
I20260929 02:10:47.806887 30140 ts_tablet_manager.cc:1397] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:10:47.807498 30142 log.cc:824] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:47.815092 30141 tablet_bootstrap.cc:492] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: No bootstrap required, opened a new log
I20260929 02:10:47.815475 30141 ts_tablet_manager.cc:1397] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260929 02:10:47.821884 30140 raft_consensus.cc:348] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.822501 30140 raft_consensus.cc:374] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:47.822669 30140 raft_consensus.cc:729] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92589f236c243d0bcc2cab354f68fd7, State: Initialized, Role: FOLLOWER
I20260929 02:10:47.823493 30140 consensus_queue.cc:260] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.825675 29992 heartbeater.cc:503] Master 127.11.182.254:45251 was elected leader, sending a full tablet report...
I20260929 02:10:47.825953 30141 raft_consensus.cc:348] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.826352 30141 raft_consensus.cc:374] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:47.826413 30140 ts_tablet_manager.cc:1428] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260929 02:10:47.826440 30141 raft_consensus.cc:729] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1006679a7cd4acda70ec44d206139c3, State: Initialized, Role: FOLLOWER
I20260929 02:10:47.826910 30142 tablet_bootstrap.cc:492] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de: No bootstrap required, opened a new log
I20260929 02:10:47.827199 30142 ts_tablet_manager.cc:1397] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.013s
I20260929 02:10:47.827083 30141 consensus_queue.cc:260] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.828280 30122 heartbeater.cc:503] Master 127.11.182.254:45251 was elected leader, sending a full tablet report...
I20260929 02:10:47.828852 30141 ts_tablet_manager.cc:1428] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 02:10:47.831691 29861 heartbeater.cc:503] Master 127.11.182.254:45251 was elected leader, sending a full tablet report...
I20260929 02:10:47.836743 30142 raft_consensus.cc:348] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.837139 30142 raft_consensus.cc:374] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:47.837244 30142 raft_consensus.cc:729] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a430f6603340c794c51122dc0962de, State: Initialized, Role: FOLLOWER
I20260929 02:10:47.837770 30142 consensus_queue.cc:260] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.839290 30142 ts_tablet_manager.cc:1428] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260929 02:10:47.920511 30148 raft_consensus.cc:482] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:10:47.920912 30148 raft_consensus.cc:504] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.922753 30148 leader_election.cc:288] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1006679a7cd4acda70ec44d206139c3 (127.11.182.195:45669), b92589f236c243d0bcc2cab354f68fd7 (127.11.182.194:36269)
I20260929 02:10:47.925858 30146 raft_consensus.cc:482] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:10:47.926148 30146 raft_consensus.cc:504] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.928078 30146 leader_election.cc:288] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7a430f6603340c794c51122dc0962de (127.11.182.193:38219), f1006679a7cd4acda70ec44d206139c3 (127.11.182.195:45669)
I20260929 02:10:47.930768 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.923214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:33471 (local address 127.11.182.195:45669)
0929 02:10:47.923455 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.923463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.923814 (+ 351us) server_negotiation.cc:407] Connection header received
0929 02:10:47.924029 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.924036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.924129 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.924228 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:47.924846 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.925586 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.927793 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.929157 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.929356 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.929464 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.930341 (+ 877us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.930487 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:10:47.931403 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.923106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45669 (local address 127.11.182.193:33471)
0929 02:10:47.923702 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:47.923718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:47.923886 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:47.924309 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:47.924316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:47.924347 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:47.924679 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.924688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.925727 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.925732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.927587 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.927599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.929940 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.929948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.930157 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.930190 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:10:47.931012 (+ 822us) client_negotiation.cc:240] Negotiation successful
0929 02:10:47.931150 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":3,"thread_start_us":344,"threads_started":1}
I20260929 02:10:47.932257 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.924001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:55171 (local address 127.11.182.194:36269)
0929 02:10:47.924214 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.924219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.924607 (+ 388us) server_negotiation.cc:407] Connection header received
0929 02:10:47.924817 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.924825 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.924888 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.924984 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:47.925516 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.926657 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.929234 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.930965 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.931092 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.931193 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.931831 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.931974 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":81}
I20260929 02:10:47.932339 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.923852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:36269 (local address 127.11.182.193:55171)
0929 02:10:47.924507 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:47.924524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:47.924668 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:47.925002 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:47.925006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:47.925028 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:47.925349 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.925357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.926813 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.926818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.929022 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.929037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.931388 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.931400 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.931631 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.931682 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 02:10:47.931774 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:10:47.931913 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":50,"thread_start_us":283,"threads_started":1}
I20260929 02:10:47.933425 29947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff79e1fb41045f2a197f4751c5fd429" candidate_uuid: "b7a430f6603340c794c51122dc0962de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92589f236c243d0bcc2cab354f68fd7" is_pre_election: true
I20260929 02:10:47.933897 29947 raft_consensus.cc:2454] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7a430f6603340c794c51122dc0962de in term 0.
I20260929 02:10:47.933939 30077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff79e1fb41045f2a197f4751c5fd429" candidate_uuid: "b7a430f6603340c794c51122dc0962de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1006679a7cd4acda70ec44d206139c3" is_pre_election: true
I20260929 02:10:47.934504 30077 raft_consensus.cc:2454] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7a430f6603340c794c51122dc0962de in term 0.
I20260929 02:10:47.934748 29749 leader_election.cc:302] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7a430f6603340c794c51122dc0962de, b92589f236c243d0bcc2cab354f68fd7; no voters:
I20260929 02:10:47.935184 30148 raft_consensus.cc:2781] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:10:47.935353 30148 raft_consensus.cc:482] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:10:47.935484 30148 raft_consensus.cc:3037] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:47.937790 30148 raft_consensus.cc:504] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.938474 30148 leader_election.cc:288] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [CANDIDATE]: Term 1 election: Requested vote from peers f1006679a7cd4acda70ec44d206139c3 (127.11.182.195:45669), b92589f236c243d0bcc2cab354f68fd7 (127.11.182.194:36269)
I20260929 02:10:47.938958 30077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff79e1fb41045f2a197f4751c5fd429" candidate_uuid: "b7a430f6603340c794c51122dc0962de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1006679a7cd4acda70ec44d206139c3"
I20260929 02:10:47.938985 29947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff79e1fb41045f2a197f4751c5fd429" candidate_uuid: "b7a430f6603340c794c51122dc0962de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92589f236c243d0bcc2cab354f68fd7"
I20260929 02:10:47.939144 30077 raft_consensus.cc:3037] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:47.939199 29947 raft_consensus.cc:3037] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:47.942107 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.928712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:45465 (local address 127.11.182.193:38219)
0929 02:10:47.928889 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.928894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.930081 (+ 1187us) server_negotiation.cc:407] Connection header received
0929 02:10:47.930320 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.930325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.930401 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.930455 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:47.933641 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.934642 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.939856 (+ 5214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.941033 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.941171 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.941246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.941751 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.941880 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:10:47.942317 30077 raft_consensus.cc:2454] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7a430f6603340c794c51122dc0962de in term 1.
I20260929 02:10:47.942515 29947 raft_consensus.cc:2454] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7a430f6603340c794c51122dc0962de in term 1.
I20260929 02:10:47.942821 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.928232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38219 (local address 127.11.182.194:45465)
0929 02:10:47.929979 (+ 1747us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:47.929998 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:47.930177 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:47.930616 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:47.930620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:47.930656 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:47.932847 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.932867 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.934829 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.934837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.939646 (+ 4809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.939662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.941369 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.941375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.941560 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.941592 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:10:47.942261 (+ 669us) client_negotiation.cc:240] Negotiation successful
0929 02:10:47.942423 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1601,"mutex_wait_us":1666,"thread_start_us":1432,"threads_started":1}
I20260929 02:10:47.942823 29749 leader_election.cc:302] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7a430f6603340c794c51122dc0962de, f1006679a7cd4acda70ec44d206139c3; no voters:
I20260929 02:10:47.943089 30148 raft_consensus.cc:2781] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:47.943681 30148 raft_consensus.cc:686] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 1 LEADER]: Becoming Leader. State: Replica: b7a430f6603340c794c51122dc0962de, State: Running, Role: LEADER
I20260929 02:10:47.943831 29816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff79e1fb41045f2a197f4751c5fd429" candidate_uuid: "b92589f236c243d0bcc2cab354f68fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a430f6603340c794c51122dc0962de" is_pre_election: true
I20260929 02:10:47.944258 30148 consensus_queue.cc:237] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } }
I20260929 02:10:47.945393 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.929023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:46155 (local address 127.11.182.195:45669)
0929 02:10:47.931022 (+ 1999us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.931027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.932113 (+ 1086us) server_negotiation.cc:407] Connection header received
0929 02:10:47.932361 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.932368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.932435 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.932507 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:47.933574 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.934744 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.941410 (+ 6666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.942693 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.942809 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.942896 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.944968 (+ 2072us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.945095 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1892,"thread_start_us":3865,"threads_started":1}
I20260929 02:10:47.945405 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.928593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45669 (local address 127.11.182.194:46155)
0929 02:10:47.932027 (+ 3434us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:47.932043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:47.932197 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:47.932692 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:47.932698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:47.932733 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:47.933420 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.933431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.935078 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.935084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.941181 (+ 6097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:47.941195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.943351 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.943357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:47.944811 (+ 1454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.944852 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 02:10:47.944943 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:10:47.945075 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3321,"mutex_wait_us":332,"thread_start_us":3112,"threads_started":1}
I20260929 02:10:47.946029 30077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff79e1fb41045f2a197f4751c5fd429" candidate_uuid: "b92589f236c243d0bcc2cab354f68fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1006679a7cd4acda70ec44d206139c3" is_pre_election: true
I20260929 02:10:47.946336 30077 raft_consensus.cc:2379] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b92589f236c243d0bcc2cab354f68fd7 in current term 1: Already voted for candidate b7a430f6603340c794c51122dc0962de in this term.
I20260929 02:10:47.946988 29880 leader_election.cc:302] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b92589f236c243d0bcc2cab354f68fd7; no voters: b7a430f6603340c794c51122dc0962de, f1006679a7cd4acda70ec44d206139c3
I20260929 02:10:47.947417 30146 raft_consensus.cc:2726] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:10:47.949066 29675 catalog_manager.cc:5495] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de reported cstate change: term changed from 0 to 1, leader changed from <none> to b7a430f6603340c794c51122dc0962de (127.11.182.193). New cstate: current_term: 1 leader_uuid: "b7a430f6603340c794c51122dc0962de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a430f6603340c794c51122dc0962de" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 } health_report { overall_health: UNKNOWN } } }
W20260929 02:10:47.962045 30123 tablet.cc:2369] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:47.970816 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:47.965622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.11.182.193:38219)
0929 02:10:47.965815 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:47.965823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:47.965844 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:10:47.965933 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:47.965937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:47.966005 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:47.966149 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:10:47.966518 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.967324 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.968783 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:47.969202 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:47.969352 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:47.969801 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:10:47.970242 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:10:47.970416 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:47.970465 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 02:10:47.970539 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:10:47.978390 30077 raft_consensus.cc:1261] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 1 FOLLOWER]: Refusing update from remote peer b7a430f6603340c794c51122dc0962de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:47.978474 29947 raft_consensus.cc:1261] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 1 FOLLOWER]: Refusing update from remote peer b7a430f6603340c794c51122dc0962de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:47.979418 30154 consensus_queue.cc:1035] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1006679a7cd4acda70ec44d206139c3" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:47.979806 30148 consensus_queue.cc:1035] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92589f236c243d0bcc2cab354f68fd7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 36269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:47.993206 30158 mvcc.cc:205] Tried to move back new op lower bound from 7334493585310294016 to 7334493585184182272. Current Snapshot: MvccSnapshot[applied={T|T < 7334493585310294016}]
I20260929 02:10:48.008366 29674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:48.008658 29674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:48.012436 29674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bf62cc5d0a4455d86220f7fefc29370: Sending DeleteTablet for 3 replicas of tablet eff79e1fb41045f2a197f4751c5fd429
I20260929 02:10:48.013494 29796 tablet_service.cc:1514] Processing DeleteTablet for tablet eff79e1fb41045f2a197f4751c5fd429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:48 UTC) from {username='slave'} at 127.0.0.1:34430
I20260929 02:10:48.013669 30057 tablet_service.cc:1514] Processing DeleteTablet for tablet eff79e1fb41045f2a197f4751c5fd429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:48 UTC) from {username='slave'} at 127.0.0.1:57476
I20260929 02:10:48.013695 29927 tablet_service.cc:1514] Processing DeleteTablet for tablet eff79e1fb41045f2a197f4751c5fd429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:48 UTC) from {username='slave'} at 127.0.0.1:52340
I20260929 02:10:48.015930 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 29734
I20260929 02:10:48.034690 30167 tablet_replica.cc:330] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de: stopping tablet replica
I20260929 02:10:48.036011 30168 tablet_replica.cc:330] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: stopping tablet replica
I20260929 02:10:48.035916 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.017267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.11.182.193:38219)
0929 02:10:48.017461 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:48.017469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:48.022287 (+ 4818us) server_negotiation.cc:407] Connection header received
0929 02:10:48.023157 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:48.023163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:48.023282 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:48.023359 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:48.024499 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:48.025817 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:48.029513 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:48.030282 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:48.030391 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:48.032917 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:48.032929 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:48.032947 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:48.033770 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:48.033855 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:48.033882 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:48.033969 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:48.035436 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0929 02:10:48.035560 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:10:48.036309 30169 tablet_replica.cc:330] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: stopping tablet replica
I20260929 02:10:48.036763 30169 raft_consensus.cc:2229] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:48.036864 29856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:48.037302 30168 raft_consensus.cc:2229] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:48.037465 30169 raft_consensus.cc:2258] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:48.038262 30168 raft_consensus.cc:2258] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:48.040467 30169 ts_tablet_manager.cc:1905] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:48.040689 30168 ts_tablet_manager.cc:1905] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:48.045732 30169 ts_tablet_manager.cc:1918] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:48.045907 30169 log.cc:1196] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/eff79e1fb41045f2a197f4751c5fd429
I20260929 02:10:48.046370 30169 ts_tablet_manager.cc:1939] T eff79e1fb41045f2a197f4751c5fd429 P f1006679a7cd4acda70ec44d206139c3: Deleting consensus metadata
I20260929 02:10:48.048002 29659 catalog_manager.cc:4841] TS f1006679a7cd4acda70ec44d206139c3 (127.11.182.195:45669): tablet eff79e1fb41045f2a197f4751c5fd429 (table test-table [id=e28d89a13ff649f48a61b8f850d9f50e]) successfully deleted
I20260929 02:10:48.048084 30168 ts_tablet_manager.cc:1918] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:48.048244 30168 log.cc:1196] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/eff79e1fb41045f2a197f4751c5fd429
I20260929 02:10:48.048797 30168 ts_tablet_manager.cc:1939] T eff79e1fb41045f2a197f4751c5fd429 P b92589f236c243d0bcc2cab354f68fd7: Deleting consensus metadata
I20260929 02:10:48.050580 29659 catalog_manager.cc:4841] TS b92589f236c243d0bcc2cab354f68fd7 (127.11.182.194:36269): tablet eff79e1fb41045f2a197f4751c5fd429 (table test-table [id=e28d89a13ff649f48a61b8f850d9f50e]) successfully deleted
I20260929 02:10:48.037784 30167 raft_consensus.cc:2229] T eff79e1fb41045f2a197f4751c5fd429 P b7a430f6603340c794c51122dc0962de [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:10:48.134816 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 29734
W20260929 02:10:48.152814 29660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.182.193:38219 (error 108)
I20260929 02:10:48.153028 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 29865
I20260929 02:10:48.154680 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.154251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38219 (local address 127.0.0.1:34468)
0929 02:10:48.154420 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:48.154536 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:10:48.154939 29660 catalog_manager.cc:4584] TS b7a430f6603340c794c51122dc0962de (127.11.182.193:38219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eff79e1fb41045f2a197f4751c5fd429: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38219: connect: Connection refused (error 111)
I20260929 02:10:48.159938 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.153850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.11.182.194:36269)
0929 02:10:48.154030 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:48.154037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:48.154180 (+ 143us) server_negotiation.cc:407] Connection header received
0929 02:10:48.154385 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:48.154390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:48.154512 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:48.154630 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:48.155112 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:48.155870 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:48.157078 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:48.157469 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:48.157564 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:48.158625 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:48.158634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:48.158647 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:48.159293 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:48.159342 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:48.159363 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:48.159452 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:48.159606 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 02:10:48.159690 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:10:48.160509 29987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:48.208616 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.208045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38219 (local address 127.0.0.1:34480)
0929 02:10:48.208287 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:48.208415 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 02:10:48.209769 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.209371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38219 (local address 127.0.0.1:34492)
0929 02:10:48.209525 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:48.209609 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:10:48.258002 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 29865
I20260929 02:10:48.270422 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.269940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38219 (local address 127.0.0.1:34502)
0929 02:10:48.270167 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:48.270260 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 02:10:48.271481 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.271122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38219 (local address 127.0.0.1:34516)
0929 02:10:48.271291 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:48.271374 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 02:10:48.275218 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 29995
I20260929 02:10:48.281695 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.275926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.11.182.195:45669)
0929 02:10:48.276088 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:48.276092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:48.276109 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:10:48.276238 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:48.276242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:48.276332 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:48.276411 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:48.276756 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:48.277421 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:48.278784 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:48.279255 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:48.279358 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:48.280440 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:48.280448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:48.280462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:48.281107 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:48.281154 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:48.281173 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:48.281258 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:48.281419 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 02:10:48.281495 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:10:48.282351 30117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:48.352229 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.351758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38219 (local address 127.0.0.1:34518)
0929 02:10:48.351973 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:48.352069 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 02:10:48.353260 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:48.352909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38219 (local address 127.0.0.1:34524)
0929 02:10:48.353081 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:48.353168 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 02:10:48.370734 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 29995
I20260929 02:10:48.387989 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 29644
I20260929 02:10:48.389066 29705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:48.497885 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 29644
2026-09-29T02:10:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2749 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:10:48 dist-test-slave-q06p krb5kdc[30178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:10:48 dist-test-slave-q06p krb5kdc[30178](info): set up 2 sockets
Sep 29 02:10:48 dist-test-slave-q06p krb5kdc[30178](info): commencing operation
krb5kdc: starting...
W20260929 02:10:49.595484 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:10:49 dist-test-slave-q06p krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:10:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:10:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:10:49.856165 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:36029
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:36029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:50.184470 30191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:50.184829 30191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:50.184998 30191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:50.192310 30191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:10:50.192473 30191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:50.192584 30191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:50.192703 30191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:10:50.192780 30191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:10:50.202494 30191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:36029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:36029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:50.203841 30191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:50.205216 30191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:50.217549 30196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:50.217527 30197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:50.217506 30199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:50.218534 30191 server_base.cc:1034] running on GCE node
I20260929 02:10:50.219416 30191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:50.220839 30191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:50.222117 30191 hybrid_clock.cc:648] HybridClock initialized: now 1790647850222052 us; error 39 us; skew 500 ppm
I20260929 02:10:50.222541 30191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:50 dist-test-slave-q06p krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647850, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:50.225960 30191 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:10:50.227880 30191 webserver.cc:460] Webserver started at http://127.11.182.254:32997/ using document root <none> and password file <none>
I20260929 02:10:50.228638 30191 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:50.228798 30191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:50.229152 30191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:50.232419 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "7f5f86ebf04549aa924a19251e6ccffe"
format_stamp: "Formatted at 2026-09-29 02:10:50 on dist-test-slave-q06p"
server_key: "aa8150040bce1100f669f522c71b5d76"
server_key_iv: "b756bac256e97a0fd0946a2e6582c262"
server_key_version: "encryptionkey@0"
I20260929 02:10:50.233256 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "7f5f86ebf04549aa924a19251e6ccffe"
format_stamp: "Formatted at 2026-09-29 02:10:50 on dist-test-slave-q06p"
server_key: "aa8150040bce1100f669f522c71b5d76"
server_key_iv: "b756bac256e97a0fd0946a2e6582c262"
server_key_version: "encryptionkey@0"
I20260929 02:10:50.238780 30191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:10:50.242350 30206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:50.243692 30191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:10:50.243896 30191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "7f5f86ebf04549aa924a19251e6ccffe"
format_stamp: "Formatted at 2026-09-29 02:10:50 on dist-test-slave-q06p"
server_key: "aa8150040bce1100f669f522c71b5d76"
server_key_iv: "b756bac256e97a0fd0946a2e6582c262"
server_key_version: "encryptionkey@0"
I20260929 02:10:50.244108 30191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:50.257699 30191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:50.262319 30191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:50.262655 30191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:50.281842 30191 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:36029
I20260929 02:10:50.281849 30257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:36029 every 8 connection(s)
I20260929 02:10:50.283156 30191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:10:50.286629 30258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:50.289601 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 30191
I20260929 02:10:50.289795 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:10:50.290252 11995 external_mini_cluster.cc:1428] Setting key 80ab7a2e21e43b2adc43df08ed31775c
I20260929 02:10:50.298758 30258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe: Bootstrap starting.
I20260929 02:10:50.302531 30258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:50.303838 30258 log.cc:824] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:10:50 dist-test-slave-q06p krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:50.306682 30258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe: No bootstrap required, opened a new log
I20260929 02:10:50.313323 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:50.292225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.11.182.254:36029)
0929 02:10:50.292976 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:50.292987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:50.293027 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:10:50.294643 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:50.294663 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:50.295122 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:50.295429 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:50.297262 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:50.298534 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:50.300349 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:50.301058 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:50.301196 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:50.306241 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:50.306248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:50.306262 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:50.306294 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:50.309375 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:50.310053 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:50.310061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:50.310071 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:50.310240 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:50.310606 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:50.310623 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:50.310625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:50.311206 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:50.311518 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:50.311920 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 02:10:50.312327 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":347,"threads_started":1}
I20260929 02:10:50.314430 30258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5f86ebf04549aa924a19251e6ccffe" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36029 } }
I20260929 02:10:50.314782 30258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:50.314908 30258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5f86ebf04549aa924a19251e6ccffe, State: Initialized, Role: FOLLOWER
I20260929 02:10:50.315521 30258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5f86ebf04549aa924a19251e6ccffe" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36029 } }
I20260929 02:10:50.315770 30258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:10:50.315928 30258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:10:50.316109 30258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:50.318182 30258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5f86ebf04549aa924a19251e6ccffe" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36029 } }
I20260929 02:10:50.318607 30258 leader_election.cc:302] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f5f86ebf04549aa924a19251e6ccffe; no voters:
I20260929 02:10:50.319166 30258 leader_election.cc:288] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:10:50.319253 30263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:50.320003 30263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [term 1 LEADER]: Becoming Leader. State: Replica: 7f5f86ebf04549aa924a19251e6ccffe, State: Running, Role: LEADER
I20260929 02:10:50.320518 30263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5f86ebf04549aa924a19251e6ccffe" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36029 } }
I20260929 02:10:50.321354 30258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:10:50.323379 30265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f5f86ebf04549aa924a19251e6ccffe. Latest consensus state: current_term: 1 leader_uuid: "7f5f86ebf04549aa924a19251e6ccffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5f86ebf04549aa924a19251e6ccffe" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36029 } } }
I20260929 02:10:50.323837 30265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:50.324414 30264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f5f86ebf04549aa924a19251e6ccffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5f86ebf04549aa924a19251e6ccffe" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 36029 } } }
I20260929 02:10:50.324723 30264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:50.327750 30271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:10:50.332228 30271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:10:50.340173 30271 catalog_manager.cc:1347] Generated new cluster ID: 0d52572fd28e448baa402e5c568996c3
I20260929 02:10:50.340288 30271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:10:50.364475 30271 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:10:50.366320 30271 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:10:50.384989 30271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe: Generated new TSK 0
I20260929 02:10:50.385807 30271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:10:50.456444 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36029
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:50.807951 30284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:50.808251 30284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:50.808426 30284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:50.815477 30284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:50.815611 30284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:50.815799 30284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:10:50.825538 30284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:50.826793 30284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:50.828224 30284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:50.841394 30292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:50.841481 30289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:50.841481 30290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:50.842357 30284 server_base.cc:1034] running on GCE node
I20260929 02:10:50.842978 30284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:50.843808 30284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:50.845018 30284 hybrid_clock.cc:648] HybridClock initialized: now 1790647850844942 us; error 56 us; skew 500 ppm
I20260929 02:10:50.845276 30284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:50 dist-test-slave-q06p krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647850, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:50.848310 30284 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:10:50.849846 30284 webserver.cc:460] Webserver started at http://127.11.182.193:40227/ using document root <none> and password file <none>
I20260929 02:10:50.850500 30284 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:50.850643 30284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:50.851019 30284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:50.853761 30284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "21155e75e9894cbab847ddcc31094f0e"
format_stamp: "Formatted at 2026-09-29 02:10:50 on dist-test-slave-q06p"
server_key: "baae4f1778b2816002c15fe369f8a67f"
server_key_iv: "328e4ca4d9ba33423f7cd0a0e7566f64"
server_key_version: "encryptionkey@0"
I20260929 02:10:50.854516 30284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "21155e75e9894cbab847ddcc31094f0e"
format_stamp: "Formatted at 2026-09-29 02:10:50 on dist-test-slave-q06p"
server_key: "baae4f1778b2816002c15fe369f8a67f"
server_key_iv: "328e4ca4d9ba33423f7cd0a0e7566f64"
server_key_version: "encryptionkey@0"
I20260929 02:10:50.859462 30284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:10:50.862908 30299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:50.864199 30284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:10:50.864390 30284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "21155e75e9894cbab847ddcc31094f0e"
format_stamp: "Formatted at 2026-09-29 02:10:50 on dist-test-slave-q06p"
server_key: "baae4f1778b2816002c15fe369f8a67f"
server_key_iv: "328e4ca4d9ba33423f7cd0a0e7566f64"
server_key_version: "encryptionkey@0"
I20260929 02:10:50.864599 30284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:50.889441 30284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:50.893573 30284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:50.893882 30284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:50.894836 30284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:50.896184 30284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:50.896260 30284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:50.896430 30284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:50.896489 30284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:50.928963 30284 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:35679
I20260929 02:10:50.928987 30411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:35679 every 8 connection(s)
I20260929 02:10:50.930284 30284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:10:50.933988 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 30284
I20260929 02:10:50.934175 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:10:50.934696 11995 external_mini_cluster.cc:1428] Setting key 9084653d5298ab4a28eb75c943d28c55
Sep 29 02:10:50 dist-test-slave-q06p krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647850, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:50.951538 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:50.932802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:55573 (local address 127.11.182.254:36029)
0929 02:10:50.933255 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:50.933261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:50.934908 (+ 1647us) server_negotiation.cc:407] Connection header received
0929 02:10:50.936389 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:50.936393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:50.936458 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:50.936581 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:50.938000 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:50.939229 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:50.941308 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:50.942047 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:50.942191 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:50.945994 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:50.945999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:50.946002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:50.946033 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:50.948581 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:50.949282 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:50.949290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:50.949293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:50.949413 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:50.949818 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:50.949825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:50.949828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:50.950084 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:50.950253 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:50.950955 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 02:10:50.951170 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":217,"threads_started":1}
I20260929 02:10:50.952428 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:50.933366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36029 (local address 127.11.182.193:55573)
0929 02:10:50.934129 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:50.934188 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:50.936107 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:50.936784 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:50.936791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:50.937282 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:50.937828 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:50.937846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:50.939380 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:50.939384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:50.941144 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:50.941160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:50.942268 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:50.942276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:50.942562 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:50.943392 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:50.943424 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:50.945782 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:50.948721 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:50.948731 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:50.948746 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:50.949156 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:50.949521 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:50.949528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:50.949532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:50.949710 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:50.950238 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:50.950246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:50.950573 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 02:10:50.950831 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 02:10:50.951363 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":328,"threads_started":1}
I20260929 02:10:50.953931 30412 heartbeater.cc:347] Connected to a master server at 127.11.182.254:36029
I20260929 02:10:50.954231 30412 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:50.955056 30412 heartbeater.cc:511] Master 127.11.182.254:36029 requested a full tablet report, sending...
I20260929 02:10:50.957152 30223 ts_manager.cc:194] Registered new tserver with Master: 21155e75e9894cbab847ddcc31094f0e (127.11.182.193:35679)
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:10:50.958657 30223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:55573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:10:50.991333 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36029
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:51.312883 30418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:51.313139 30418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:51.313290 30418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:51.320390 30418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:51.320488 30418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:51.320683 30418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:10:51.330024 30418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:51.331235 30418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:51.332490 30418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:51.345506 30423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:51.345530 30426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:51.345475 30424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:51.346525 30418 server_base.cc:1034] running on GCE node
I20260929 02:10:51.347257 30418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:51.348312 30418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:51.349555 30418 hybrid_clock.cc:648] HybridClock initialized: now 1790647851349468 us; error 69 us; skew 500 ppm
I20260929 02:10:51.349892 30418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:51 dist-test-slave-q06p krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647851, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:51.359129 30418 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:10:51.360471 30418 webserver.cc:460] Webserver started at http://127.11.182.194:44083/ using document root <none> and password file <none>
I20260929 02:10:51.361122 30418 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:51.361265 30418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:51.361559 30418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:51.364539 30418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "1ac7b0ad26014cd7a1e183491502b2e8"
format_stamp: "Formatted at 2026-09-29 02:10:51 on dist-test-slave-q06p"
server_key: "9ede5fe34cac501015df29d616c04f83"
server_key_iv: "f9bd0212aea8f8b6185d50b1c3cf77ca"
server_key_version: "encryptionkey@0"
I20260929 02:10:51.365305 30418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "1ac7b0ad26014cd7a1e183491502b2e8"
format_stamp: "Formatted at 2026-09-29 02:10:51 on dist-test-slave-q06p"
server_key: "9ede5fe34cac501015df29d616c04f83"
server_key_iv: "f9bd0212aea8f8b6185d50b1c3cf77ca"
server_key_version: "encryptionkey@0"
I20260929 02:10:51.370458 30418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:10:51.373728 30433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:51.375154 30418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:10:51.375334 30418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "1ac7b0ad26014cd7a1e183491502b2e8"
format_stamp: "Formatted at 2026-09-29 02:10:51 on dist-test-slave-q06p"
server_key: "9ede5fe34cac501015df29d616c04f83"
server_key_iv: "f9bd0212aea8f8b6185d50b1c3cf77ca"
server_key_version: "encryptionkey@0"
I20260929 02:10:51.375555 30418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:51.407203 30418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:51.411793 30418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:51.412119 30418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:51.413179 30418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:51.414536 30418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:51.414618 30418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:51.414736 30418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:51.414786 30418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:51.448848 30418 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:45061
I20260929 02:10:51.448875 30545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:45061 every 8 connection(s)
I20260929 02:10:51.450197 30418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:10:51.455416 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 30418
I20260929 02:10:51.455619 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:10:51.456168 11995 external_mini_cluster.cc:1428] Setting key b4f475c966867a3a3ff503fc3cea65a9
Sep 29 02:10:51 dist-test-slave-q06p krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647851, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:51.471993 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:51.452775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:56859 (local address 127.11.182.254:36029)
0929 02:10:51.453524 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:51.453529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:51.454395 (+ 866us) server_negotiation.cc:407] Connection header received
0929 02:10:51.456863 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:51.456869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:51.456966 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:51.457056 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:51.458696 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.459925 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.461455 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.462209 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.462360 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:51.465699 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:51.465708 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:51.465716 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:51.465762 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:51.468703 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:51.469675 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:51.469680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:51.469682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:51.469817 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:51.470300 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:51.470303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:51.470305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:51.470544 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:51.470685 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:51.471537 (+ 852us) server_negotiation.cc:299] Negotiation successful
0929 02:10:51.471722 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":221,"threads_started":1}
I20260929 02:10:51.473559 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:51.453449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36029 (local address 127.11.182.194:56859)
0929 02:10:51.454224 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:51.454277 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:51.456590 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:51.457332 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:51.457340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:51.457899 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:51.458485 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:51.458502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.460068 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.460072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:51.461308 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:51.461320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.462422 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.462426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:51.462614 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:51.463355 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:51.463380 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:51.465446 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:51.468833 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:51.468845 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:51.468864 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:51.469530 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:51.469938 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:51.469945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:51.469952 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:51.470180 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:51.470672 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:51.470682 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:51.471149 (+ 467us) client_negotiation.cc:769] Sending connection context
0929 02:10:51.471459 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 02:10:51.472019 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":311,"threads_started":1}
I20260929 02:10:51.475062 30546 heartbeater.cc:347] Connected to a master server at 127.11.182.254:36029
I20260929 02:10:51.475376 30546 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:51.476104 30546 heartbeater.cc:511] Master 127.11.182.254:36029 requested a full tablet report, sending...
I20260929 02:10:51.477671 30223 ts_manager.cc:194] Registered new tserver with Master: 1ac7b0ad26014cd7a1e183491502b2e8 (127.11.182.194:45061)
I20260929 02:10:51.478564 30223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:56859
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:10:51.513737 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36029
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:51.832615 30552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:51.832937 30552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:51.833106 30552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:51.840464 30552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:51.840618 30552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:51.840801 30552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:10:51.850879 30552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:36029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:51.852236 30552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:51.853569 30552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:51.866734 30557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:51.866724 30560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:51.866729 30558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:51.867563 30552 server_base.cc:1034] running on GCE node
I20260929 02:10:51.868240 30552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:51.869315 30552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:51.870534 30552 hybrid_clock.cc:648] HybridClock initialized: now 1790647851870464 us; error 68 us; skew 500 ppm
I20260929 02:10:51.870924 30552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:51 dist-test-slave-q06p krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647851, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:51.874055 30552 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:10:51.875802 30552 webserver.cc:460] Webserver started at http://127.11.182.195:40515/ using document root <none> and password file <none>
I20260929 02:10:51.876452 30552 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:51.876554 30552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:51.876816 30552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:51.879993 30552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "709103a022654b1484a0ee61a57c90f1"
format_stamp: "Formatted at 2026-09-29 02:10:51 on dist-test-slave-q06p"
server_key: "9434782f80bdc26c21ad5c74a22c9b5d"
server_key_iv: "a73a59e2c0919adbd0d88dc65a29c898"
server_key_version: "encryptionkey@0"
I20260929 02:10:51.880688 30552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "709103a022654b1484a0ee61a57c90f1"
format_stamp: "Formatted at 2026-09-29 02:10:51 on dist-test-slave-q06p"
server_key: "9434782f80bdc26c21ad5c74a22c9b5d"
server_key_iv: "a73a59e2c0919adbd0d88dc65a29c898"
server_key_version: "encryptionkey@0"
I20260929 02:10:51.885797 30552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:10:51.889199 30567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:51.890551 30552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:10:51.890791 30552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "709103a022654b1484a0ee61a57c90f1"
format_stamp: "Formatted at 2026-09-29 02:10:51 on dist-test-slave-q06p"
server_key: "9434782f80bdc26c21ad5c74a22c9b5d"
server_key_iv: "a73a59e2c0919adbd0d88dc65a29c898"
server_key_version: "encryptionkey@0"
I20260929 02:10:51.891045 30552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:51.902637 30552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:51.906461 30552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:51.906741 30552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:51.907696 30552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:51.908921 30552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:51.908988 30552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:51.909175 30552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:51.909225 30552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:51.943387 30552 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:34415
I20260929 02:10:51.943428 30679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:34415 every 8 connection(s)
I20260929 02:10:51.944648 30552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:10:51.945093 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 30552
I20260929 02:10:51.945297 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:10:51.945876 11995 external_mini_cluster.cc:1428] Setting key be1e5205aa97e8460b87765e8806b177
Sep 29 02:10:51 dist-test-slave-q06p krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647851, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:51.961588 30412 heartbeater.cc:503] Master 127.11.182.254:36029 was elected leader, sending a full tablet report...
I20260929 02:10:51.964648 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:51.948518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:40901 (local address 127.11.182.254:36029)
0929 02:10:51.948707 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:51.948713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:51.949166 (+ 453us) server_negotiation.cc:407] Connection header received
0929 02:10:51.950909 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:51.950917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:51.950988 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:51.951066 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:51.952532 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.953221 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.954812 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.955253 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.955355 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:51.958604 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:51.958609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:51.958612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:51.958643 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:51.961716 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:51.962552 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:51.962556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:51.962558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:51.962674 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:51.963211 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:51.963215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:51.963219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:51.963501 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:51.963612 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:51.964264 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 02:10:51.964431 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:10:51.965600 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:51.948020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:36029 (local address 127.11.182.195:40901)
0929 02:10:51.948984 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:51.949027 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:51.950663 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:51.951267 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:51.951274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:51.951767 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:51.952392 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:51.952410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.953395 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.953400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:51.954669 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:51.954688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.955465 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.955469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:51.955678 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:51.956294 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:51.956315 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:51.958411 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:51.961872 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:51.961882 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:51.961898 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:51.962441 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:51.962829 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:51.962835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:51.962838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:51.963103 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:51.963625 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:51.963640 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:51.963947 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 02:10:51.964177 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 02:10:51.964555 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":312,"threads_started":1}
I20260929 02:10:51.966977 30680 heartbeater.cc:347] Connected to a master server at 127.11.182.254:36029
I20260929 02:10:51.967240 30680 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:51.967921 30680 heartbeater.cc:511] Master 127.11.182.254:36029 requested a full tablet report, sending...
I20260929 02:10:51.969317 30223 ts_manager.cc:194] Registered new tserver with Master: 709103a022654b1484a0ee61a57c90f1 (127.11.182.195:34415)
I20260929 02:10:51.970139 30223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:40901
I20260929 02:10:51.973497 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:10:51.989943 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:51.977399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.11.182.254:36029)
0929 02:10:51.977630 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:51.977636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:51.977859 (+ 223us) server_negotiation.cc:407] Connection header received
0929 02:10:51.977986 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:51.977989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:51.978047 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:51.978122 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:51.979578 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.980408 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.981617 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:51.982010 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:51.982103 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:51.984668 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:51.984672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:51.984676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:51.984705 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:51.987325 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:51.988051 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:51.988054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:51.988056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:51.988166 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:51.988564 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:51.988567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:51.988569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:51.988843 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:51.989047 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:51.989430 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 02:10:51.989562 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:10:51.993924 30223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:10:52.000941 30223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:10:52.036698 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.013403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.11.182.193:35679)
0929 02:10:52.028005 (+ 14602us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:52.028014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:52.028046 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:10:52.028164 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:52.028173 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:52.028440 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:52.028638 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:52.029463 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.032109 (+ 2646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.034199 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.035256 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.035460 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.035653 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:52.036216 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 02:10:52.036374 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14384,"thread_start_us":14237,"threads_started":1}
I20260929 02:10:52.037029 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.013547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.11.182.195:34415)
0929 02:10:52.027593 (+ 14046us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:52.027617 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:52.028134 (+ 517us) server_negotiation.cc:407] Connection header received
0929 02:10:52.028416 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:52.028424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:52.028716 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:52.028935 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:52.030019 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.031700 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.034302 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.035724 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.035931 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.036095 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:52.036495 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 02:10:52.036666 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13870,"thread_start_us":13708,"threads_started":1}
I20260929 02:10:52.037449 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.013279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:34415 (local address 127.0.0.1:40334)
0929 02:10:52.028023 (+ 14744us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:52.028041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:52.028230 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:52.029240 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:52.029245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:52.029286 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:52.029843 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.029856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.031826 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.031830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.034058 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.034070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.035941 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.035946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.036105 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.036133 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:10:52.036214 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:10:52.036343 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14631,"mutex_wait_us":32,"thread_start_us":13107,"threads_started":1}
I20260929 02:10:52.038066 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.013060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:35679 (local address 127.0.0.1:35086)
0929 02:10:52.026411 (+ 13351us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:52.026468 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:52.026638 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:52.028671 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:52.028677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:52.028713 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:52.029260 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.029276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.032229 (+ 2953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.032237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.034020 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.034043 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.035460 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.035466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.035737 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.035779 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:10:52.035946 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 02:10:52.036130 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13074,"mutex_wait_us":96,"thread_start_us":14701,"threads_started":1}
I20260929 02:10:52.038887 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.028165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:45061 (local address 127.0.0.1:48784)
0929 02:10:52.029750 (+ 1585us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:52.029782 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:52.029906 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:52.030593 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:52.030598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:52.030655 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:52.031697 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.031710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.033521 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.033526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.035787 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.035803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.038048 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.038058 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.038235 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.038266 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:10:52.038361 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:10:52.038501 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1446,"mutex_wait_us":15,"thread_start_us":578,"threads_started":1}
I20260929 02:10:52.039827 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.028411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.11.182.194:45061)
0929 02:10:52.029367 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:52.029377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:52.029894 (+ 517us) server_negotiation.cc:407] Connection header received
0929 02:10:52.030106 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:52.030111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:52.030254 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:52.030459 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:52.031898 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.033379 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.035971 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.037441 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.039081 (+ 1640us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.039265 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:52.039342 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 02:10:52.039502 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":664,"threads_started":1}
I20260929 02:10:52.041560 30347 tablet_service.cc:1467] Processing CreateTablet for tablet cf48e3e4b51945e5940843d9e184b1cc (DEFAULT_TABLE table=test-table [id=4479721066994cd594331a6d0c39eb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:52.041587 30615 tablet_service.cc:1467] Processing CreateTablet for tablet cf48e3e4b51945e5940843d9e184b1cc (DEFAULT_TABLE table=test-table [id=4479721066994cd594331a6d0c39eb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:52.042887 30347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf48e3e4b51945e5940843d9e184b1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:52.043038 30615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf48e3e4b51945e5940843d9e184b1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:52.043192 30481 tablet_service.cc:1467] Processing CreateTablet for tablet cf48e3e4b51945e5940843d9e184b1cc (DEFAULT_TABLE table=test-table [id=4479721066994cd594331a6d0c39eb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:52.044687 30481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf48e3e4b51945e5940843d9e184b1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:52.054754 30698 tablet_bootstrap.cc:492] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: Bootstrap starting.
I20260929 02:10:52.057687 30699 tablet_bootstrap.cc:492] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: Bootstrap starting.
I20260929 02:10:52.058209 30698 tablet_bootstrap.cc:654] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:52.059677 30698 log.cc:824] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:52.061383 30699 tablet_bootstrap.cc:654] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:52.061697 30700 tablet_bootstrap.cc:492] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: Bootstrap starting.
I20260929 02:10:52.062719 30699 log.cc:824] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:52.063354 30698 tablet_bootstrap.cc:492] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: No bootstrap required, opened a new log
I20260929 02:10:52.063628 30698 ts_tablet_manager.cc:1397] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260929 02:10:52.065115 30700 tablet_bootstrap.cc:654] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:52.065963 30699 tablet_bootstrap.cc:492] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: No bootstrap required, opened a new log
I20260929 02:10:52.066211 30699 ts_tablet_manager.cc:1397] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:10:52.066255 30700 log.cc:824] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:52.069285 30700 tablet_bootstrap.cc:492] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: No bootstrap required, opened a new log
I20260929 02:10:52.069525 30700 ts_tablet_manager.cc:1397] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:10:52.073402 30699 raft_consensus.cc:348] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } }
I20260929 02:10:52.073772 30699 raft_consensus.cc:374] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:52.073894 30699 raft_consensus.cc:729] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709103a022654b1484a0ee61a57c90f1, State: Initialized, Role: FOLLOWER
I20260929 02:10:52.074052 30698 raft_consensus.cc:348] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } }
I20260929 02:10:52.074466 30698 raft_consensus.cc:374] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:52.074574 30698 raft_consensus.cc:729] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21155e75e9894cbab847ddcc31094f0e, State: Initialized, Role: FOLLOWER
I20260929 02:10:52.074528 30699 consensus_queue.cc:260] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } }
I20260929 02:10:52.075205 30698 consensus_queue.cc:260] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } }
I20260929 02:10:52.076942 30698 ts_tablet_manager.cc:1428] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260929 02:10:52.079454 30700 raft_consensus.cc:348] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } }
I20260929 02:10:52.079790 30700 raft_consensus.cc:374] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:52.079900 30700 raft_consensus.cc:729] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac7b0ad26014cd7a1e183491502b2e8, State: Initialized, Role: FOLLOWER
I20260929 02:10:52.080380 30700 consensus_queue.cc:260] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } }
I20260929 02:10:52.081498 30546 heartbeater.cc:503] Master 127.11.182.254:36029 was elected leader, sending a full tablet report...
I20260929 02:10:52.081944 30700 ts_tablet_manager.cc:1428] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260929 02:10:52.083923 30680 heartbeater.cc:503] Master 127.11.182.254:36029 was elected leader, sending a full tablet report...
I20260929 02:10:52.084537 30699 ts_tablet_manager.cc:1428] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
W20260929 02:10:52.182639 30413 tablet.cc:2369] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:52.193288 30705 raft_consensus.cc:482] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:10:52.193614 30705 raft_consensus.cc:504] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } }
I20260929 02:10:52.195571 30705 leader_election.cc:288] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 709103a022654b1484a0ee61a57c90f1 (127.11.182.195:34415), 1ac7b0ad26014cd7a1e183491502b2e8 (127.11.182.194:45061)
W20260929 02:10:52.196410 30681 tablet.cc:2369] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:52.202294 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.196077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:49643 (local address 127.11.182.194:45061)
0929 02:10:52.196335 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:52.196341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:52.196894 (+ 553us) server_negotiation.cc:407] Connection header received
0929 02:10:52.197171 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:52.197177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:52.197250 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:52.197347 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:52.197873 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.198600 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.200411 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.201339 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.201512 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.201585 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:52.201914 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 02:10:52.202034 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 02:10:52.202407 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.196079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:42069 (local address 127.11.182.195:34415)
0929 02:10:52.196488 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:52.196495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:52.197351 (+ 856us) server_negotiation.cc:407] Connection header received
0929 02:10:52.197412 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:52.197415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:52.197467 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:52.197565 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:52.198168 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.198954 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.200619 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.201446 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.201627 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.201707 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:52.202042 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 02:10:52.202162 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260929 02:10:52.202442 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.195911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:45061 (local address 127.11.182.193:49643)
0929 02:10:52.196775 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:52.196800 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:52.197022 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:52.197360 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:52.197365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:52.197387 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:52.197725 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.197734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.198725 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.198732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.200253 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.200273 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.201537 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.201543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.201723 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.201786 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 02:10:52.201888 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 02:10:52.202013 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":26,"thread_start_us":876,"threads_started":1}
W20260929 02:10:52.202521 30547 tablet.cc:2369] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:52.202922 30707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.196045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:34415 (local address 127.11.182.193:42069)
0929 02:10:52.196998 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:52.197018 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:52.197171 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:52.197597 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:52.197601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:52.197629 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:52.198008 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.198017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.199096 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.199111 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.200461 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:52.200474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.201670 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.201676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:52.201878 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.201908 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:10:52.202006 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:10:52.202137 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":55,"thread_start_us":475,"threads_started":1}
I20260929 02:10:52.203871 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf48e3e4b51945e5940843d9e184b1cc" candidate_uuid: "21155e75e9894cbab847ddcc31094f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" is_pre_election: true
I20260929 02:10:52.203871 30635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf48e3e4b51945e5940843d9e184b1cc" candidate_uuid: "21155e75e9894cbab847ddcc31094f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709103a022654b1484a0ee61a57c90f1" is_pre_election: true
I20260929 02:10:52.204432 30501 raft_consensus.cc:2454] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21155e75e9894cbab847ddcc31094f0e in term 0.
I20260929 02:10:52.204429 30635 raft_consensus.cc:2454] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21155e75e9894cbab847ddcc31094f0e in term 0.
I20260929 02:10:52.205219 30300 leader_election.cc:302] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac7b0ad26014cd7a1e183491502b2e8, 21155e75e9894cbab847ddcc31094f0e; no voters:
I20260929 02:10:52.205686 30705 raft_consensus.cc:2781] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:10:52.205878 30705 raft_consensus.cc:482] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:10:52.206010 30705 raft_consensus.cc:3037] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:52.208323 30705 raft_consensus.cc:504] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } }
I20260929 02:10:52.208880 30705 leader_election.cc:288] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [CANDIDATE]: Term 1 election: Requested vote from peers 709103a022654b1484a0ee61a57c90f1 (127.11.182.195:34415), 1ac7b0ad26014cd7a1e183491502b2e8 (127.11.182.194:45061)
I20260929 02:10:52.209312 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf48e3e4b51945e5940843d9e184b1cc" candidate_uuid: "21155e75e9894cbab847ddcc31094f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac7b0ad26014cd7a1e183491502b2e8"
I20260929 02:10:52.209326 30635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf48e3e4b51945e5940843d9e184b1cc" candidate_uuid: "21155e75e9894cbab847ddcc31094f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709103a022654b1484a0ee61a57c90f1"
I20260929 02:10:52.209506 30501 raft_consensus.cc:3037] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:52.209510 30635 raft_consensus.cc:3037] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:52.211798 30635 raft_consensus.cc:2454] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21155e75e9894cbab847ddcc31094f0e in term 1.
I20260929 02:10:52.212155 30501 raft_consensus.cc:2454] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21155e75e9894cbab847ddcc31094f0e in term 1.
I20260929 02:10:52.212191 30302 leader_election.cc:302] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21155e75e9894cbab847ddcc31094f0e, 709103a022654b1484a0ee61a57c90f1; no voters:
I20260929 02:10:52.212476 30705 raft_consensus.cc:2781] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:52.212932 30705 raft_consensus.cc:686] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 1 LEADER]: Becoming Leader. State: Replica: 21155e75e9894cbab847ddcc31094f0e, State: Running, Role: LEADER
I20260929 02:10:52.213471 30705 consensus_queue.cc:237] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } }
I20260929 02:10:52.217751 30223 catalog_manager.cc:5495] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 21155e75e9894cbab847ddcc31094f0e (127.11.182.193). New cstate: current_term: 1 leader_uuid: "21155e75e9894cbab847ddcc31094f0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21155e75e9894cbab847ddcc31094f0e" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 } health_report { overall_health: UNKNOWN } } }
I20260929 02:10:52.264382 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.259249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.11.182.193:35679)
0929 02:10:52.259430 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:52.259435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:52.259528 (+ 93us) server_negotiation.cc:407] Connection header received
0929 02:10:52.259699 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:52.259703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:52.259780 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:52.259903 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:10:52.260238 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.260930 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.262346 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.262751 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.262909 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.263416 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:10:52.263888 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:10:52.264013 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:52.264143 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 02:10:52.264182 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:10:52.272418 30501 raft_consensus.cc:1261] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [term 1 FOLLOWER]: Refusing update from remote peer 21155e75e9894cbab847ddcc31094f0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:52.272418 30635 raft_consensus.cc:1261] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [term 1 FOLLOWER]: Refusing update from remote peer 21155e75e9894cbab847ddcc31094f0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:52.273334 30709 consensus_queue.cc:1035] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "709103a022654b1484a0ee61a57c90f1" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:52.273664 30705 consensus_queue.cc:1035] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ac7b0ad26014cd7a1e183491502b2e8" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 45061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:52.283124 30712 mvcc.cc:205] Tried to move back new op lower bound from 7334493602898411520 to 7334493602670088192. Current Snapshot: MvccSnapshot[applied={T|T < 7334493602898411520}]
I20260929 02:10:52.298971 30714 mvcc.cc:205] Tried to move back new op lower bound from 7334493602898411520 to 7334493602670088192. Current Snapshot: MvccSnapshot[applied={T|T < 7334493602898411520}]
I20260929 02:10:52.302795 30223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:52.303156 30223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:52.307399 30713 mvcc.cc:205] Tried to move back new op lower bound from 7334493602898411520 to 7334493602670088192. Current Snapshot: MvccSnapshot[applied={T|T < 7334493602898411520 or (T in {7334493602898411520})}]
I20260929 02:10:52.308597 30481 tablet_service.cc:1514] Processing DeleteTablet for tablet cf48e3e4b51945e5940843d9e184b1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:52 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:48784
I20260929 02:10:52.309288 30223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f5f86ebf04549aa924a19251e6ccffe: Sending DeleteTablet for 3 replicas of tablet cf48e3e4b51945e5940843d9e184b1cc
I20260929 02:10:52.309414 30722 tablet_replica.cc:330] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: stopping tablet replica
I20260929 02:10:52.310029 30722 raft_consensus.cc:2229] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:52.310184 30347 tablet_service.cc:1514] Processing DeleteTablet for tablet cf48e3e4b51945e5940843d9e184b1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:52 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:35086
I20260929 02:10:52.310539 30722 raft_consensus.cc:2258] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:52.312572 30722 ts_tablet_manager.cc:1905] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:52.314383 30481 tablet_service.cc:1514] Processing DeleteTablet for tablet cf48e3e4b51945e5940843d9e184b1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:52 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:48784
I20260929 02:10:52.314488 30615 tablet_service.cc:1514] Processing DeleteTablet for tablet cf48e3e4b51945e5940843d9e184b1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:52 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:40334
I20260929 02:10:52.318565 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 30284
I20260929 02:10:52.319319 30723 tablet_replica.cc:330] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: stopping tablet replica
I20260929 02:10:52.319741 30723 raft_consensus.cc:2229] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:10:52.320328 30723 raft_consensus.cc:2258] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:52.321280 30722 ts_tablet_manager.cc:1918] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:52.321493 30722 log.cc:1196] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/cf48e3e4b51945e5940843d9e184b1cc
I20260929 02:10:52.322163 30722 ts_tablet_manager.cc:1939] T cf48e3e4b51945e5940843d9e184b1cc P 1ac7b0ad26014cd7a1e183491502b2e8: Deleting consensus metadata
I20260929 02:10:52.322846 30723 ts_tablet_manager.cc:1905] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:10:52.325385 30207 catalog_manager.cc:4808] TS 1ac7b0ad26014cd7a1e183491502b2e8 (127.11.182.194:45061): delete failed for tablet cf48e3e4b51945e5940843d9e184b1cc because the tablet was not found. No further retry: Not found: Tablet not found: cf48e3e4b51945e5940843d9e184b1cc
I20260929 02:10:52.326968 30724 tablet_replica.cc:330] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: stopping tablet replica
I20260929 02:10:52.327371 30724 raft_consensus.cc:2229] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:52.327836 30724 raft_consensus.cc:2258] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:52.329725 30724 ts_tablet_manager.cc:1905] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:52.330677 30723 ts_tablet_manager.cc:1918] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:52.330827 30723 log.cc:1196] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/cf48e3e4b51945e5940843d9e184b1cc
I20260929 02:10:52.331449 30723 ts_tablet_manager.cc:1939] T cf48e3e4b51945e5940843d9e184b1cc P 21155e75e9894cbab847ddcc31094f0e: Deleting consensus metadata
I20260929 02:10:52.333050 30207 catalog_manager.cc:4841] TS 21155e75e9894cbab847ddcc31094f0e (127.11.182.193:35679): tablet cf48e3e4b51945e5940843d9e184b1cc (table test-table [id=4479721066994cd594331a6d0c39eb3f]) successfully deleted
I20260929 02:10:52.335693 30724 ts_tablet_manager.cc:1918] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:52.335852 30724 log.cc:1196] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/cf48e3e4b51945e5940843d9e184b1cc
I20260929 02:10:52.336392 30724 ts_tablet_manager.cc:1939] T cf48e3e4b51945e5940843d9e184b1cc P 709103a022654b1484a0ee61a57c90f1: Deleting consensus metadata
I20260929 02:10:52.337779 30209 catalog_manager.cc:4841] TS 709103a022654b1484a0ee61a57c90f1 (127.11.182.195:34415): tablet cf48e3e4b51945e5940843d9e184b1cc (table test-table [id=4479721066994cd594331a6d0c39eb3f]) successfully deleted
Sep 29 02:10:52 dist-test-slave-q06p krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:10:52.350946 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.319590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.11.182.193:35679)
0929 02:10:52.319774 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:52.319779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:52.333084 (+ 13305us) server_negotiation.cc:407] Connection header received
0929 02:10:52.333528 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:52.333536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:52.333619 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:52.333730 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:52.335396 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.336695 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.338632 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.339289 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.339405 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.344451 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:52.344463 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:52.344475 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:52.344508 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:52.347619 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:52.348404 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:52.348414 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:52.348417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:52.348569 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:52.348986 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:52.348989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:52.348991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:52.349519 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:52.349798 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:52.350271 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 02:10:52.350508 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 02:10:52.351614 30407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:52.449266 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 30284
I20260929 02:10:52.470999 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 30418
Sep 29 02:10:52 dist-test-slave-q06p krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:10:52.485877 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.471836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.11.182.194:45061)
0929 02:10:52.472052 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:52.472058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:52.472079 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:10:52.472148 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:52.472155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:52.472240 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:52.472355 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:52.473600 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.474453 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.475788 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.476179 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.476304 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.480457 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:52.480475 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:52.480489 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:52.480520 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:52.483140 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:52.483786 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:52.483801 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:52.483805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:52.483921 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:52.484298 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:52.484305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:52.484308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:52.484683 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:52.484977 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:52.485289 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 02:10:52.485507 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 02:10:52.486564 30541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:52.576401 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 30418
I20260929 02:10:52.594141 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 30552
Sep 29 02:10:52 dist-test-slave-q06p krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:10:52.608520 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:52.594931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.11.182.195:34415)
0929 02:10:52.595102 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:52.595107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:52.595186 (+ 79us) server_negotiation.cc:407] Connection header received
0929 02:10:52.595329 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:52.595334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:52.595455 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:52.595556 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:52.596818 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.597617 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.598811 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:52.599264 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:52.599356 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:52.603631 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:52.603646 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:52.603662 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:52.603692 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:52.606138 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:52.606700 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:52.606706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:52.606708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:52.606933 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:52.607348 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:52.607351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:52.607353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:52.607660 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:52.607897 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:52.608122 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 02:10:52.608273 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:10:52.609124 30675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:52.698349 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 30552
I20260929 02:10:52.715086 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 30191
I20260929 02:10:52.716207 30253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:52.826222 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 30191
2026-09-29T02:10:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4335 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T02:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:10:52Z Disabled control of system clock
I20260929 02:10:52.909025 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:46337
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:39517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:10:53.229261 30733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:53.229640 30733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:53.229797 30733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:53.237236 30733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:10:53.237401 30733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:53.237496 30733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:53.237588 30733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:10:53.237668 30733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:10:53.246703 30733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:46337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:53.248026 30733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:53.249771 30733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:53.264621 30738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:53.264545 30739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:53.264580 30741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:53.265033 30733 server_base.cc:1034] running on GCE node
I20260929 02:10:53.265832 30733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:53.267349 30733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:53.268571 30733 hybrid_clock.cc:648] HybridClock initialized: now 1790647853268540 us; error 50 us; skew 500 ppm
I20260929 02:10:53.268930 30733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:10:53.271154 30733 webserver.cc:460] Webserver started at http://127.11.182.254:42687/ using document root <none> and password file <none>
I20260929 02:10:53.271822 30733 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:53.271951 30733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:53.272274 30733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:53.275229 30733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c"
format_stamp: "Formatted at 2026-09-29 02:10:53 on dist-test-slave-q06p"
server_key: "1364fb210a9244dd81636291a073e4d7"
server_key_iv: "5f0afaa6dbfe654ebdb3c9b85cf80fd9"
server_key_version: "encryptionkey@0"
I20260929 02:10:53.276028 30733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c"
format_stamp: "Formatted at 2026-09-29 02:10:53 on dist-test-slave-q06p"
server_key: "1364fb210a9244dd81636291a073e4d7"
server_key_iv: "5f0afaa6dbfe654ebdb3c9b85cf80fd9"
server_key_version: "encryptionkey@0"
I20260929 02:10:53.280954 30733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:10:53.284308 30747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:53.285530 30733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:10:53.285709 30733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c"
format_stamp: "Formatted at 2026-09-29 02:10:53 on dist-test-slave-q06p"
server_key: "1364fb210a9244dd81636291a073e4d7"
server_key_iv: "5f0afaa6dbfe654ebdb3c9b85cf80fd9"
server_key_version: "encryptionkey@0"
I20260929 02:10:53.285912 30733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:53.302523 30733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:53.303382 30733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:53.303661 30733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:53.321587 30733 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:46337
I20260929 02:10:53.321643 30798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:46337 every 8 connection(s)
I20260929 02:10:53.322937 30733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:10:53.326481 30799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:53.329744 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 30733
I20260929 02:10:53.329916 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:10:53.330453 11995 external_mini_cluster.cc:1428] Setting key 394ed10b20b86ef7ab4948bb8a59cefd
I20260929 02:10:53.338304 30799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c: Bootstrap starting.
I20260929 02:10:53.341473 30799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:53.342571 30799 log.cc:824] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:53.343525 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:53.332444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.11.182.254:46337)
0929 02:10:53.333364 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:53.333374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:53.333412 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:10:53.335129 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:53.335149 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:53.335586 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:53.335960 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:53.336531 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:53.337746 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:53.339178 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:53.339875 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:53.340069 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:53.341202 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:53.341210 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:53.341222 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:53.341778 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:53.341873 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:53.341893 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:53.342015 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:53.342232 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 02:10:53.342522 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":326,"threads_started":1}
I20260929 02:10:53.345640 30799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c: No bootstrap required, opened a new log
I20260929 02:10:53.353191 30799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46337 } }
I20260929 02:10:53.353509 30799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:53.353580 30799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d72a6de8f8c4e30b054e0f72dd1c33c, State: Initialized, Role: FOLLOWER
I20260929 02:10:53.354238 30799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46337 } }
I20260929 02:10:53.354511 30799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:10:53.354696 30799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:10:53.354887 30799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:53.356974 30799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46337 } }
I20260929 02:10:53.357451 30799 leader_election.cc:302] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d72a6de8f8c4e30b054e0f72dd1c33c; no voters:
I20260929 02:10:53.358069 30799 leader_election.cc:288] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:10:53.358168 30804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:53.359460 30804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [term 1 LEADER]: Becoming Leader. State: Replica: 6d72a6de8f8c4e30b054e0f72dd1c33c, State: Running, Role: LEADER
I20260929 02:10:53.359933 30804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46337 } }
I20260929 02:10:53.360188 30799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:10:53.362895 30805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46337 } } }
I20260929 02:10:53.362874 30806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d72a6de8f8c4e30b054e0f72dd1c33c. Latest consensus state: current_term: 1 leader_uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d72a6de8f8c4e30b054e0f72dd1c33c" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46337 } } }
I20260929 02:10:53.363826 30805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:53.363919 30806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:53.366243 30811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:10:53.371433 30811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:10:53.380898 30811 catalog_manager.cc:1347] Generated new cluster ID: 8d0b899b69f34668b61c869f0fc168f5
I20260929 02:10:53.381031 30811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:10:53.432170 30811 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:10:53.433467 30811 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:10:53.443614 30811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c: Generated new TSK 0
I20260929 02:10:53.444461 30811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:10:53.454047 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46337
--builtin_ntp_servers=127.11.182.212:39517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:10:53.779791 30823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:53.780040 30823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:53.780191 30823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:53.787258 30823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:53.787395 30823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:53.787573 30823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:10:53.796613 30823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:53.797904 30823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:53.799690 30823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:53.812300 30828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:53.812366 30831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:53.812399 30829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:53.813112 30823 server_base.cc:1034] running on GCE node
I20260929 02:10:53.813640 30823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:53.814482 30823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:53.815681 30823 hybrid_clock.cc:648] HybridClock initialized: now 1790647853815649 us; error 34 us; skew 500 ppm
I20260929 02:10:53.815944 30823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:10:53.817992 30823 webserver.cc:460] Webserver started at http://127.11.182.193:44015/ using document root <none> and password file <none>
I20260929 02:10:53.818621 30823 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:53.818710 30823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:53.819070 30823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:53.821758 30823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "7fd086d2ea4e43d399f97afb2ac193ab"
format_stamp: "Formatted at 2026-09-29 02:10:53 on dist-test-slave-q06p"
server_key: "4f4387143167cc916d6a0bbd812bca45"
server_key_iv: "ce4af511c6ae8bddbb27eb8350198a99"
server_key_version: "encryptionkey@0"
I20260929 02:10:53.822474 30823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "7fd086d2ea4e43d399f97afb2ac193ab"
format_stamp: "Formatted at 2026-09-29 02:10:53 on dist-test-slave-q06p"
server_key: "4f4387143167cc916d6a0bbd812bca45"
server_key_iv: "ce4af511c6ae8bddbb27eb8350198a99"
server_key_version: "encryptionkey@0"
I20260929 02:10:53.827335 30823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:10:53.830493 30837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:53.831962 30823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:10:53.832139 30823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "7fd086d2ea4e43d399f97afb2ac193ab"
format_stamp: "Formatted at 2026-09-29 02:10:53 on dist-test-slave-q06p"
server_key: "4f4387143167cc916d6a0bbd812bca45"
server_key_iv: "ce4af511c6ae8bddbb27eb8350198a99"
server_key_version: "encryptionkey@0"
I20260929 02:10:53.832357 30823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:53.858289 30823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:53.859226 30823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:53.859534 30823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:53.860575 30823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:53.862174 30823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:53.862304 30823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:53.862428 30823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:53.862517 30823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:53.896185 30823 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:45969
I20260929 02:10:53.896271 30949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:45969 every 8 connection(s)
I20260929 02:10:53.897500 30823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:10:53.899324 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 30823
I20260929 02:10:53.899497 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:10:53.899986 11995 external_mini_cluster.cc:1428] Setting key 6569ad3e1b4de6bb47402197ab01e06f
I20260929 02:10:53.904367 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46337
--builtin_ntp_servers=127.11.182.212:39517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:10:53.911393 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:53.901520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:50309 (local address 127.11.182.254:46337)
0929 02:10:53.903221 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:53.903226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:53.903248 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:10:53.904168 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:53.904177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:53.904257 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:53.904390 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:53.905474 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:53.906687 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:53.908252 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:53.908980 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:53.909120 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:53.910397 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:53.910406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:53.910414 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:53.910439 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:53.910494 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:53.910512 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:53.910607 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:53.910805 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 02:10:53.910936 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1588,"thread_start_us":1494,"threads_started":1}
I20260929 02:10:53.912184 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:53.900978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46337 (local address 127.11.182.193:50309)
0929 02:10:53.902033 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:53.902083 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:53.903900 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:53.904627 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:53.904637 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:53.904747 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:53.905296 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:53.905314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:53.906823 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:53.906827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:53.908107 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:53.908118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:53.909175 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:53.909180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:53.909382 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:53.910085 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:10:53.910087 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:53.910125 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:10:53.910211 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:53.910608 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:53.910616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:53.910621 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:10:53.910759 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 02:10:53.911108 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
I20260929 02:10:53.913721 30950 heartbeater.cc:347] Connected to a master server at 127.11.182.254:46337
I20260929 02:10:53.914041 30950 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:53.914752 30950 heartbeater.cc:511] Master 127.11.182.254:46337 requested a full tablet report, sending...
I20260929 02:10:53.916731 30764 ts_manager.cc:194] Registered new tserver with Master: 7fd086d2ea4e43d399f97afb2ac193ab (127.11.182.193:45969)
I20260929 02:10:53.918260 30764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.193:50309
W20260929 02:10:54.223027 30954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:54.223320 30954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:54.223485 30954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:54.230616 30954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:54.230747 30954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:54.230984 30954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:10:54.240118 30954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:54.241339 30954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:54.243112 30954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:54.255352 30959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:54.255427 30960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:54.255396 30962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:54.256425 30954 server_base.cc:1034] running on GCE node
I20260929 02:10:54.257043 30954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:54.258021 30954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:54.259264 30954 hybrid_clock.cc:648] HybridClock initialized: now 1790647854259220 us; error 29 us; skew 500 ppm
I20260929 02:10:54.259586 30954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:10:54.262603 30954 webserver.cc:460] Webserver started at http://127.11.182.194:46653/ using document root <none> and password file <none>
I20260929 02:10:54.263334 30954 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:54.263458 30954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:54.263764 30954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:54.266568 30954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "e37501925d824b2ab42239fc5243946f"
format_stamp: "Formatted at 2026-09-29 02:10:54 on dist-test-slave-q06p"
server_key: "56aa654c2897d91384ce920951ab1842"
server_key_iv: "3b17902d4938151cba544451c400c17d"
server_key_version: "encryptionkey@0"
I20260929 02:10:54.267354 30954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "e37501925d824b2ab42239fc5243946f"
format_stamp: "Formatted at 2026-09-29 02:10:54 on dist-test-slave-q06p"
server_key: "56aa654c2897d91384ce920951ab1842"
server_key_iv: "3b17902d4938151cba544451c400c17d"
server_key_version: "encryptionkey@0"
I20260929 02:10:54.272025 30954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:10:54.275579 30968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:54.276880 30954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:10:54.277077 30954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "e37501925d824b2ab42239fc5243946f"
format_stamp: "Formatted at 2026-09-29 02:10:54 on dist-test-slave-q06p"
server_key: "56aa654c2897d91384ce920951ab1842"
server_key_iv: "3b17902d4938151cba544451c400c17d"
server_key_version: "encryptionkey@0"
I20260929 02:10:54.277288 30954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:54.304934 30954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:54.305755 30954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:54.306061 30954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:54.306986 30954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:54.308385 30954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:54.308509 30954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:54.308653 30954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:54.308748 30954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:54.342203 30954 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:42775
I20260929 02:10:54.342283 31080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:42775 every 8 connection(s)
I20260929 02:10:54.343541 30954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:10:54.349134 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 30954
I20260929 02:10:54.349355 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:10:54.349849 11995 external_mini_cluster.cc:1428] Setting key 7c804f6602bdf339aee4b8237b813268
I20260929 02:10:54.352906 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46337
--builtin_ntp_servers=127.11.182.212:39517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:10:54.355978 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.346209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:51093 (local address 127.11.182.254:46337)
0929 02:10:54.346391 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:54.346396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:54.347810 (+ 1414us) server_negotiation.cc:407] Connection header received
0929 02:10:54.349631 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:54.349637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:54.349726 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:54.349844 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:54.351032 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.351825 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.353377 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.353765 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.353901 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.355199 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:54.355209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:54.355212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:54.355228 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:54.355257 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:54.355270 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:54.355361 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:54.355581 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 02:10:54.355673 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:10:54.356886 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.346686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46337 (local address 127.11.182.194:51093)
0929 02:10:54.347667 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:54.347699 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:54.349347 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:54.350064 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:54.350072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:54.350154 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:54.350807 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.350830 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.351975 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.351980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.353232 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.353243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.353978 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.353985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.354203 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.354903 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:10:54.354908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:54.354960 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:10:54.355027 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:54.355361 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:54.355374 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:54.355383 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:10:54.355558 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 02:10:54.355915 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":401,"threads_started":1}
I20260929 02:10:54.358237 31081 heartbeater.cc:347] Connected to a master server at 127.11.182.254:46337
I20260929 02:10:54.358532 31081 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:54.359272 31081 heartbeater.cc:511] Master 127.11.182.254:46337 requested a full tablet report, sending...
I20260929 02:10:54.360714 30764 ts_manager.cc:194] Registered new tserver with Master: e37501925d824b2ab42239fc5243946f (127.11.182.194:42775)
I20260929 02:10:54.362049 30764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.194:51093
W20260929 02:10:54.677855 31084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:54.678117 31084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:54.678200 31084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:54.685413 31084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:54.685504 31084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:54.685611 31084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:10:54.694446 31084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:46337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:54.695659 31084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:54.697407 31084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:54.710450 31089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:54.710376 31092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:54.710397 31090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:54.711354 31084 server_base.cc:1034] running on GCE node
I20260929 02:10:54.711863 31084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:54.712677 31084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:54.713924 31084 hybrid_clock.cc:648] HybridClock initialized: now 1790647854713889 us; error 47 us; skew 500 ppm
I20260929 02:10:54.714236 31084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:10:54.716290 31084 webserver.cc:460] Webserver started at http://127.11.182.195:34985/ using document root <none> and password file <none>
I20260929 02:10:54.716941 31084 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:54.717068 31084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:54.717366 31084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:54.720122 31084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "95aa35c11f144ca69bd76d3bbbef995a"
format_stamp: "Formatted at 2026-09-29 02:10:54 on dist-test-slave-q06p"
server_key: "53e1677c8779d375f733a9a078d33a25"
server_key_iv: "3f5e99967bcae5c8ecd2d8e649e048a9"
server_key_version: "encryptionkey@0"
I20260929 02:10:54.720902 31084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "95aa35c11f144ca69bd76d3bbbef995a"
format_stamp: "Formatted at 2026-09-29 02:10:54 on dist-test-slave-q06p"
server_key: "53e1677c8779d375f733a9a078d33a25"
server_key_iv: "3f5e99967bcae5c8ecd2d8e649e048a9"
server_key_version: "encryptionkey@0"
I20260929 02:10:54.725679 31084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 02:10:54.728897 31098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:54.730049 31084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:10:54.730224 31084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "95aa35c11f144ca69bd76d3bbbef995a"
format_stamp: "Formatted at 2026-09-29 02:10:54 on dist-test-slave-q06p"
server_key: "53e1677c8779d375f733a9a078d33a25"
server_key_iv: "3f5e99967bcae5c8ecd2d8e649e048a9"
server_key_version: "encryptionkey@0"
I20260929 02:10:54.730427 31084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:54.761457 31084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:54.762341 31084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:54.762647 31084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:54.763636 31084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:54.765090 31084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:54.765218 31084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:54.765360 31084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:54.765470 31084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:54.799358 31084 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:34561
I20260929 02:10:54.799445 31210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:34561 every 8 connection(s)
I20260929 02:10:54.800621 31084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:10:54.808460 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 31084
I20260929 02:10:54.808662 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:10:54.809119 11995 external_mini_cluster.cc:1428] Setting key 79cb4d56ad53f95fdd19838a52f9100f
I20260929 02:10:54.813069 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.803140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:52137 (local address 127.11.182.254:46337)
0929 02:10:54.803399 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:54.803406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:54.804745 (+ 1339us) server_negotiation.cc:407] Connection header received
0929 02:10:54.806402 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:54.806407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:54.806473 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:54.806557 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:54.807593 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.808548 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.810254 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.810873 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.810995 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.812457 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:54.812461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:54.812464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:54.812481 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:54.812510 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:54.812521 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:54.812587 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:54.812772 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 02:10:54.812844 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:10:54.814010 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.803754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46337 (local address 127.11.182.195:52137)
0929 02:10:54.804589 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:54.804623 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:54.806164 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:54.806748 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:54.806758 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:54.806837 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:54.807457 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.807473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.808725 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.808732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.810101 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.810117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.811106 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.811113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.811406 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.812171 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:10:54.812174 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:54.812212 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:10:54.812293 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:54.812601 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:54.812608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:54.812613 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:10:54.812742 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 02:10:54.813009 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":253,"threads_started":1}
I20260929 02:10:54.815291 31211 heartbeater.cc:347] Connected to a master server at 127.11.182.254:46337
I20260929 02:10:54.815567 31211 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:54.816217 31211 heartbeater.cc:511] Master 127.11.182.254:46337 requested a full tablet report, sending...
I20260929 02:10:54.817461 30764 ts_manager.cc:194] Registered new tserver with Master: 95aa35c11f144ca69bd76d3bbbef995a (127.11.182.195:34561)
I20260929 02:10:54.818171 30764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.195:52137
I20260929 02:10:54.823892 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:10:54.833554 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.827693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.11.182.254:46337)
0929 02:10:54.827913 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:54.827922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:54.828276 (+ 354us) server_negotiation.cc:407] Connection header received
0929 02:10:54.828466 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:54.828471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:54.828547 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:54.828650 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:54.829102 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.829781 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.831158 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.831548 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.831671 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.832803 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:54.832807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:54.832811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:54.832827 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:54.832855 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:54.832866 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:54.833012 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:54.833207 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 02:10:54.833259 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:10:54.837308 30764 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:10:54.843809 30764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:10:54.864696 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.854706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45969 (local address 127.0.0.1:45566)
0929 02:10:54.855501 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:54.855543 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:54.855713 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:54.857046 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:54.857053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:54.857081 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:54.857557 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.857570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.859801 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.859806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.861868 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.861886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.863711 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.863720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.863932 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.863987 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 02:10:54.864120 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 02:10:54.864311 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":1,"thread_start_us":381,"threads_started":1}
I20260929 02:10:54.864745 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.855057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.11.182.193:45969)
0929 02:10:54.856359 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:54.856367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:54.856397 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:10:54.856492 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:54.856502 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:54.856796 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:54.857762 (+ 966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:54.858029 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.859649 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.862104 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.863517 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.863679 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.863898 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:54.864190 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 02:10:54.864387 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":994,"threads_started":1}
I20260929 02:10:54.865624 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.855360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:42775 (local address 127.0.0.1:56610)
0929 02:10:54.856609 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:54.856627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:54.856754 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:54.858177 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:54.858182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:54.858204 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:54.858698 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.858710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.860480 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.860485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.862667 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.862682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.864738 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.864745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.864934 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.864967 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:10:54.865101 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:10:54.865234 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"mutex_wait_us":23,"thread_start_us":988,"threads_started":1}
I20260929 02:10:54.866386 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.856010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.11.182.194:42775)
0929 02:10:54.857242 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:54.857254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:54.857282 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:10:54.857374 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:54.857380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:54.857618 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:54.857900 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:54.858910 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.860335 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.862908 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.864252 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.864397 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.864586 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:54.865880 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0929 02:10:54.866048 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":930,"threads_started":1}
I20260929 02:10:54.866401 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.855978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:34561 (local address 127.0.0.1:42040)
0929 02:10:54.858416 (+ 2438us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:54.858434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:54.858552 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:54.859426 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:54.859431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:54.859453 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:54.859827 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.859863 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.861811 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.861816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.863583 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:54.863616 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.865598 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.865607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:54.865827 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.865857 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:10:54.865935 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:10:54.866054 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2284,"mutex_wait_us":3,"thread_start_us":2119,"threads_started":1}
I20260929 02:10:54.866752 31228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:54.856560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.11.182.195:34561)
0929 02:10:54.857947 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:54.857963 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:54.858682 (+ 719us) server_negotiation.cc:407] Connection header received
0929 02:10:54.858784 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:54.858790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:54.859013 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:54.859178 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:54.860053 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.861646 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.863832 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:54.865402 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:54.865583 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:54.865970 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:54.866037 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 02:10:54.866186 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":712,"threads_started":1}
I20260929 02:10:54.869468 30885 tablet_service.cc:1467] Processing CreateTablet for tablet c31175c2a6c64ae58f71cf262018d62e (DEFAULT_TABLE table=test-table [id=5db63d96334749f58c7a427d9da605c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:54.870817 30885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31175c2a6c64ae58f71cf262018d62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:54.873487 31016 tablet_service.cc:1467] Processing CreateTablet for tablet c31175c2a6c64ae58f71cf262018d62e (DEFAULT_TABLE table=test-table [id=5db63d96334749f58c7a427d9da605c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:54.874617 31016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31175c2a6c64ae58f71cf262018d62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:54.874171 31146 tablet_service.cc:1467] Processing CreateTablet for tablet c31175c2a6c64ae58f71cf262018d62e (DEFAULT_TABLE table=test-table [id=5db63d96334749f58c7a427d9da605c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:54.875581 31146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31175c2a6c64ae58f71cf262018d62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:54.885842 31229 tablet_bootstrap.cc:492] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab: Bootstrap starting.
I20260929 02:10:54.889415 31231 tablet_bootstrap.cc:492] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: Bootstrap starting.
I20260929 02:10:54.891248 31229 tablet_bootstrap.cc:654] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:54.892323 31231 tablet_bootstrap.cc:654] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:54.892791 31229 log.cc:824] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:54.893298 31231 log.cc:824] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:54.895177 31230 tablet_bootstrap.cc:492] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: Bootstrap starting.
I20260929 02:10:54.896644 31231 tablet_bootstrap.cc:492] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: No bootstrap required, opened a new log
I20260929 02:10:54.896906 31231 ts_tablet_manager.cc:1397] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 02:10:54.897689 31229 tablet_bootstrap.cc:492] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab: No bootstrap required, opened a new log
I20260929 02:10:54.897949 31229 ts_tablet_manager.cc:1397] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260929 02:10:54.899063 31230 tablet_bootstrap.cc:654] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:54.900358 31230 log.cc:824] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:54.905977 31231 raft_consensus.cc:348] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } }
I20260929 02:10:54.906342 31231 raft_consensus.cc:374] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:54.906437 31231 raft_consensus.cc:729] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95aa35c11f144ca69bd76d3bbbef995a, State: Initialized, Role: FOLLOWER
I20260929 02:10:54.907238 31231 consensus_queue.cc:260] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } }
I20260929 02:10:54.908329 31211 heartbeater.cc:503] Master 127.11.182.254:46337 was elected leader, sending a full tablet report...
I20260929 02:10:54.908954 31231 ts_tablet_manager.cc:1428] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 02:10:54.909104 31230 tablet_bootstrap.cc:492] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: No bootstrap required, opened a new log
I20260929 02:10:54.909361 31230 ts_tablet_manager.cc:1397] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260929 02:10:54.912082 31229 raft_consensus.cc:348] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } }
I20260929 02:10:54.912599 31229 raft_consensus.cc:374] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:54.912753 31229 raft_consensus.cc:729] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd086d2ea4e43d399f97afb2ac193ab, State: Initialized, Role: FOLLOWER
I20260929 02:10:54.913403 31229 consensus_queue.cc:260] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } }
I20260929 02:10:54.916338 30950 heartbeater.cc:503] Master 127.11.182.254:46337 was elected leader, sending a full tablet report...
I20260929 02:10:54.916920 31229 ts_tablet_manager.cc:1428] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab: Time spent starting tablet: real 0.019s user 0.011s sys 0.007s
I20260929 02:10:54.919118 31230 raft_consensus.cc:348] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } }
I20260929 02:10:54.919548 31230 raft_consensus.cc:374] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:54.919653 31230 raft_consensus.cc:729] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37501925d824b2ab42239fc5243946f, State: Initialized, Role: FOLLOWER
I20260929 02:10:54.920207 31230 consensus_queue.cc:260] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } }
I20260929 02:10:54.921190 31081 heartbeater.cc:503] Master 127.11.182.254:46337 was elected leader, sending a full tablet report...
I20260929 02:10:54.921761 31230 ts_tablet_manager.cc:1428] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 02:10:55.039264 31236 raft_consensus.cc:482] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:10:55.039585 31236 raft_consensus.cc:504] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } }
I20260929 02:10:55.041366 31236 leader_election.cc:288] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95aa35c11f144ca69bd76d3bbbef995a (127.11.182.195:34561), e37501925d824b2ab42239fc5243946f (127.11.182.194:42775)
I20260929 02:10:55.048123 31228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.041676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:42647 (local address 127.11.182.195:34561)
0929 02:10:55.041871 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:55.041877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:55.042589 (+ 712us) server_negotiation.cc:407] Connection header received
0929 02:10:55.042774 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:55.042780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:55.042887 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:55.042987 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:55.043580 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.044371 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.046375 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.047208 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.047382 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:55.047451 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:55.047765 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 02:10:55.047887 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:10:55.048146 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.041922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:49455 (local address 127.11.182.194:42775)
0929 02:10:55.042085 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:55.042089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:55.042795 (+ 706us) server_negotiation.cc:407] Connection header received
0929 02:10:55.042995 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:55.043002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:55.043099 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:55.043200 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:55.043821 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.044522 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.046415 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.047197 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.047362 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:55.047442 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:55.047794 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 02:10:55.047910 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:10:55.048220 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.041546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:34561 (local address 127.11.182.193:42647)
0929 02:10:55.042449 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:55.042472 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:55.042620 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:55.043016 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:55.043021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:55.043044 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:55.043419 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:55.043433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.044507 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.044512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:55.046171 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:55.046183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.047418 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.047424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:55.047580 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:55.047603 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:10:55.047673 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 02:10:55.047818 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":82,"thread_start_us":537,"threads_started":1}
I20260929 02:10:55.048607 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.041857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:42775 (local address 127.11.182.193:49455)
0929 02:10:55.042655 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:55.042688 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:55.042823 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:55.043226 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:55.043229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:55.043248 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:55.043631 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:55.043643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.044663 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.044666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:55.046174 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:55.046185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.047382 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.047388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:55.047607 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:55.047630 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:10:55.047741 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:10:55.047862 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":82,"thread_start_us":443,"threads_started":1}
I20260929 02:10:55.049321 31166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31175c2a6c64ae58f71cf262018d62e" candidate_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95aa35c11f144ca69bd76d3bbbef995a" is_pre_election: true
I20260929 02:10:55.049525 31036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31175c2a6c64ae58f71cf262018d62e" candidate_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37501925d824b2ab42239fc5243946f" is_pre_election: true
I20260929 02:10:55.049916 31166 raft_consensus.cc:2454] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd086d2ea4e43d399f97afb2ac193ab in term 0.
I20260929 02:10:55.050037 31036 raft_consensus.cc:2454] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd086d2ea4e43d399f97afb2ac193ab in term 0.
I20260929 02:10:55.050662 30841 leader_election.cc:302] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd086d2ea4e43d399f97afb2ac193ab, e37501925d824b2ab42239fc5243946f; no voters:
I20260929 02:10:55.051092 31236 raft_consensus.cc:2781] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:10:55.051244 31236 raft_consensus.cc:482] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:10:55.051373 31236 raft_consensus.cc:3037] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:10:55.052173 31212 tablet.cc:2369] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:55.053649 31236 raft_consensus.cc:504] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } }
I20260929 02:10:55.054342 31236 leader_election.cc:288] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [CANDIDATE]: Term 1 election: Requested vote from peers 95aa35c11f144ca69bd76d3bbbef995a (127.11.182.195:34561), e37501925d824b2ab42239fc5243946f (127.11.182.194:42775)
I20260929 02:10:55.054734 31166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31175c2a6c64ae58f71cf262018d62e" candidate_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95aa35c11f144ca69bd76d3bbbef995a"
I20260929 02:10:55.054966 31166 raft_consensus.cc:3037] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:55.054983 31036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31175c2a6c64ae58f71cf262018d62e" candidate_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37501925d824b2ab42239fc5243946f"
I20260929 02:10:55.055158 31036 raft_consensus.cc:3037] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:55.057066 31166 raft_consensus.cc:2454] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd086d2ea4e43d399f97afb2ac193ab in term 1.
I20260929 02:10:55.057386 31036 raft_consensus.cc:2454] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd086d2ea4e43d399f97afb2ac193ab in term 1.
I20260929 02:10:55.057471 30840 leader_election.cc:302] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd086d2ea4e43d399f97afb2ac193ab, 95aa35c11f144ca69bd76d3bbbef995a; no voters:
I20260929 02:10:55.057780 31236 raft_consensus.cc:2781] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:55.058210 31236 raft_consensus.cc:686] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [term 1 LEADER]: Becoming Leader. State: Replica: 7fd086d2ea4e43d399f97afb2ac193ab, State: Running, Role: LEADER
I20260929 02:10:55.058763 31236 consensus_queue.cc:237] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } }
I20260929 02:10:55.063271 30764 catalog_manager.cc:5495] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd086d2ea4e43d399f97afb2ac193ab (127.11.182.193). New cstate: current_term: 1 leader_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd086d2ea4e43d399f97afb2ac193ab" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 45969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 } health_report { overall_health: UNKNOWN } } }
W20260929 02:10:55.095494 31082 tablet.cc:2369] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:55.112251 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.106834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.11.182.193:45969)
0929 02:10:55.107095 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:55.107104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:55.107126 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:10:55.107243 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:55.107247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:55.107330 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:55.107483 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:10:55.107893 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.108579 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.109980 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.110384 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.110512 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:55.111190 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:10:55.111668 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:10:55.111850 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:55.111963 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 02:10:55.112020 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:10:55.120581 31166 raft_consensus.cc:1261] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [term 1 FOLLOWER]: Refusing update from remote peer 7fd086d2ea4e43d399f97afb2ac193ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:55.120581 31036 raft_consensus.cc:1261] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [term 1 FOLLOWER]: Refusing update from remote peer 7fd086d2ea4e43d399f97afb2ac193ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:55.121649 31240 consensus_queue.cc:1035] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e37501925d824b2ab42239fc5243946f" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:55.122017 31236 consensus_queue.cc:1035] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "95aa35c11f144ca69bd76d3bbbef995a" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 34561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:55.135229 31243 mvcc.cc:205] Tried to move back new op lower bound from 7334493614562922496 to 7334493614324953088. Current Snapshot: MvccSnapshot[applied={T|T < 7334493614562922496}]
I20260929 02:10:55.139801 31244 mvcc.cc:205] Tried to move back new op lower bound from 7334493614562922496 to 7334493614324953088. Current Snapshot: MvccSnapshot[applied={T|T < 7334493614562922496}]
I20260929 02:10:55.150017 30763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:55.150295 30763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:55.153679 30763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d72a6de8f8c4e30b054e0f72dd1c33c: Sending DeleteTablet for 3 replicas of tablet c31175c2a6c64ae58f71cf262018d62e
I20260929 02:10:55.154676 30885 tablet_service.cc:1514] Processing DeleteTablet for tablet c31175c2a6c64ae58f71cf262018d62e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:55 UTC) from {username='slave'} at 127.0.0.1:45566
I20260929 02:10:55.155063 31016 tablet_service.cc:1514] Processing DeleteTablet for tablet c31175c2a6c64ae58f71cf262018d62e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:55 UTC) from {username='slave'} at 127.0.0.1:56610
I20260929 02:10:55.155532 31146 tablet_service.cc:1514] Processing DeleteTablet for tablet c31175c2a6c64ae58f71cf262018d62e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:55 UTC) from {username='slave'} at 127.0.0.1:42040
I20260929 02:10:55.156425 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 30823
I20260929 02:10:55.177354 31254 tablet_replica.cc:330] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: stopping tablet replica
I20260929 02:10:55.177825 31254 raft_consensus.cc:2229] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:55.178318 31254 raft_consensus.cc:2258] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:55.180279 31255 tablet_replica.cc:330] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: stopping tablet replica
I20260929 02:10:55.180703 31255 raft_consensus.cc:2229] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:55.180982 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.158062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.11.182.193:45969)
0929 02:10:55.158296 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:55.158319 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:55.168942 (+ 10623us) server_negotiation.cc:407] Connection header received
0929 02:10:55.169460 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:55.169474 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:55.169599 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:55.169697 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:55.170487 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.172104 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.174676 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.175650 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.175774 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:55.179137 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:55.179150 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:55.179169 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:55.180002 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:55.180063 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:55.180087 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:55.180202 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:55.180481 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 02:10:55.180599 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:10:55.181006 31254 ts_tablet_manager.cc:1905] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:55.181198 31255 raft_consensus.cc:2258] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:55.182323 30945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:55.183236 31255 ts_tablet_manager.cc:1905] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:55.187984 31254 ts_tablet_manager.cc:1918] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:55.188158 31254 log.cc:1196] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/c31175c2a6c64ae58f71cf262018d62e
I20260929 02:10:55.188663 31254 ts_tablet_manager.cc:1939] T c31175c2a6c64ae58f71cf262018d62e P e37501925d824b2ab42239fc5243946f: Deleting consensus metadata
I20260929 02:10:55.190315 30751 catalog_manager.cc:4841] TS e37501925d824b2ab42239fc5243946f (127.11.182.194:42775): tablet c31175c2a6c64ae58f71cf262018d62e (table test-table [id=5db63d96334749f58c7a427d9da605c2]) successfully deleted
I20260929 02:10:55.191385 31255 ts_tablet_manager.cc:1918] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:55.191541 31255 log.cc:1196] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/c31175c2a6c64ae58f71cf262018d62e
I20260929 02:10:55.192080 31255 ts_tablet_manager.cc:1939] T c31175c2a6c64ae58f71cf262018d62e P 95aa35c11f144ca69bd76d3bbbef995a: Deleting consensus metadata
I20260929 02:10:55.193830 30750 catalog_manager.cc:4841] TS 95aa35c11f144ca69bd76d3bbbef995a (127.11.182.195:34561): tablet c31175c2a6c64ae58f71cf262018d62e (table test-table [id=5db63d96334749f58c7a427d9da605c2]) successfully deleted
I20260929 02:10:55.279165 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 30823
I20260929 02:10:55.279196 31253 tablet_replica.cc:330] T c31175c2a6c64ae58f71cf262018d62e P 7fd086d2ea4e43d399f97afb2ac193ab: stopping tablet replica
I20260929 02:10:55.297631 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 30954
W20260929 02:10:55.297614 30750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.182.193:45969 (error 108)
I20260929 02:10:55.299669 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.299243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45969 (local address 127.0.0.1:60124)
0929 02:10:55.299415 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:55.299531 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:10:55.299906 30750 catalog_manager.cc:4584] TS 7fd086d2ea4e43d399f97afb2ac193ab (127.11.182.193:45969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c31175c2a6c64ae58f71cf262018d62e: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45969: connect: Connection refused (error 111)
I20260929 02:10:55.304772 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.298405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.11.182.194:42775)
0929 02:10:55.298586 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:55.298592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:55.298742 (+ 150us) server_negotiation.cc:407] Connection header received
0929 02:10:55.298965 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:55.298971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:55.299079 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:55.299191 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:55.299566 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.300463 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.301680 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.302084 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.302223 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:55.303353 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:55.303370 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:55.303383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:55.304000 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:55.304067 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:55.304091 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:55.304202 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:55.304345 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 02:10:55.304432 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:10:55.305444 31076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:55.353454 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.352871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45969 (local address 127.0.0.1:60130)
0929 02:10:55.353132 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:55.353248 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 02:10:55.354624 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.354204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45969 (local address 127.0.0.1:60136)
0929 02:10:55.354371 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:55.354464 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:10:55.393214 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 30954
I20260929 02:10:55.410463 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 31084
I20260929 02:10:55.415166 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.414694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45969 (local address 127.0.0.1:60140)
0929 02:10:55.414927 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:55.415002 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 02:10:55.416186 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.415855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45969 (local address 127.0.0.1:60156)
0929 02:10:55.415999 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:55.416079 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 02:10:55.417831 31228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.411383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49216 (local address 127.11.182.195:34561)
0929 02:10:55.411597 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:55.411605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:55.411627 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:10:55.411843 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:55.411848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:55.411935 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:55.412050 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:55.412480 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.413211 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.414641 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:55.415145 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:55.415296 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:55.416465 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:55.416474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:55.416489 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:10:55.417101 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:55.417159 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:10:55.417178 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:55.417281 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:55.417416 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 02:10:55.417504 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:10:55.418517 31206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:55.496866 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.496339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45969 (local address 127.0.0.1:60168)
0929 02:10:55.496554 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:55.496703 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 02:10:55.497951 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:55.497531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:45969 (local address 127.0.0.1:60182)
0929 02:10:55.497767 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:55.497842 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:45969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 02:10:55.508126 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 31084
I20260929 02:10:55.524744 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 30733
I20260929 02:10:55.525930 30794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:55.637192 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 30733
2026-09-29T02:10:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2802 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:10:55 dist-test-slave-q06p krb5kdc[31264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:10:55 dist-test-slave-q06p krb5kdc[31264](info): set up 2 sockets
Sep 29 02:10:55 dist-test-slave-q06p krb5kdc[31264](info): commencing operation
krb5kdc: starting...
W20260929 02:10:56.735472 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:10:56 dist-test-slave-q06p krb5kdc[31264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:10:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:10:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:10:56.978368 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:35377
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:35377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:57.300741 31277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:57.301034 31277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:57.301175 31277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:57.308420 31277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:10:57.308545 31277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:57.308637 31277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:57.308755 31277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:10:57.308841 31277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:10:57.318537 31277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:35377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:35377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:57.319862 31277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:57.321146 31277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:57.332522 31283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:57.333055 31282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:57.332782 31285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:57.333899 31277 server_base.cc:1034] running on GCE node
I20260929 02:10:57.334540 31277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:57.335882 31277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:57.337096 31277 hybrid_clock.cc:648] HybridClock initialized: now 1790647857337031 us; error 51 us; skew 500 ppm
I20260929 02:10:57.337438 31277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:57 dist-test-slave-q06p krb5kdc[31264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647857, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:57.344157 31277 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:10:57.345583 31277 webserver.cc:460] Webserver started at http://127.11.182.254:46779/ using document root <none> and password file <none>
I20260929 02:10:57.346314 31277 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:57.346446 31277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:57.346753 31277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:57.349720 31277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "53f69e57b3954aebaabc7b4591beda2a"
format_stamp: "Formatted at 2026-09-29 02:10:57 on dist-test-slave-q06p"
server_key: "f0086a88035e8a941ea3b01eaefefd69"
server_key_iv: "5fcd9f36ed4380b60243bbe39c3c4e26"
server_key_version: "encryptionkey@0"
I20260929 02:10:57.350489 31277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "53f69e57b3954aebaabc7b4591beda2a"
format_stamp: "Formatted at 2026-09-29 02:10:57 on dist-test-slave-q06p"
server_key: "f0086a88035e8a941ea3b01eaefefd69"
server_key_iv: "5fcd9f36ed4380b60243bbe39c3c4e26"
server_key_version: "encryptionkey@0"
I20260929 02:10:57.355420 31277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 02:10:57.358594 31292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:57.359943 31277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:10:57.360137 31277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "53f69e57b3954aebaabc7b4591beda2a"
format_stamp: "Formatted at 2026-09-29 02:10:57 on dist-test-slave-q06p"
server_key: "f0086a88035e8a941ea3b01eaefefd69"
server_key_iv: "5fcd9f36ed4380b60243bbe39c3c4e26"
server_key_version: "encryptionkey@0"
I20260929 02:10:57.360352 31277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:57.377308 31277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:57.381672 31277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:57.381975 31277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:57.399721 31277 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:35377
I20260929 02:10:57.399747 31343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:35377 every 8 connection(s)
I20260929 02:10:57.401055 31277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:10:57.404739 31344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:57.410493 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 31277
I20260929 02:10:57.410687 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:10:57.411151 11995 external_mini_cluster.cc:1428] Setting key da2240a22974a0be34899a3484d4d743
I20260929 02:10:57.415172 31344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a: Bootstrap starting.
I20260929 02:10:57.418807 31344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:57.420238 31344 log.cc:824] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:57.423497 31344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a: No bootstrap required, opened a new log
Sep 29 02:10:57 dist-test-slave-q06p krb5kdc[31264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:57.432627 31344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f69e57b3954aebaabc7b4591beda2a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 35377 } }
I20260929 02:10:57.433053 31344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:57.433182 31344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f69e57b3954aebaabc7b4591beda2a, State: Initialized, Role: FOLLOWER
I20260929 02:10:57.433832 31344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f69e57b3954aebaabc7b4591beda2a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 35377 } }
I20260929 02:10:57.434141 31344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:10:57.434307 31344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:10:57.434499 31344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:57.436693 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:57.413183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.11.182.254:35377)
0929 02:10:57.414032 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:57.414043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:57.414088 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:10:57.416034 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:57.416060 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:57.416456 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:57.416823 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:57.418659 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:57.420798 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:57.422566 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:57.423314 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:57.423464 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:57.428430 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:57.428438 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:57.428457 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:57.428508 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:57.432107 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:57.432866 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:57.432875 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:57.432882 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:57.433184 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:57.433592 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:57.433599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:57.433603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:57.434302 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:57.434648 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:57.435151 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 02:10:57.435609 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":408,"threads_started":1}
I20260929 02:10:57.437252 31344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f69e57b3954aebaabc7b4591beda2a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 35377 } }
I20260929 02:10:57.437727 31344 leader_election.cc:302] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53f69e57b3954aebaabc7b4591beda2a; no voters:
I20260929 02:10:57.438398 31344 leader_election.cc:288] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:10:57.438534 31349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:57.440515 31349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [term 1 LEADER]: Becoming Leader. State: Replica: 53f69e57b3954aebaabc7b4591beda2a, State: Running, Role: LEADER
I20260929 02:10:57.440768 31344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:10:57.441094 31349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f69e57b3954aebaabc7b4591beda2a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 35377 } }
I20260929 02:10:57.444558 31351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53f69e57b3954aebaabc7b4591beda2a. Latest consensus state: current_term: 1 leader_uuid: "53f69e57b3954aebaabc7b4591beda2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f69e57b3954aebaabc7b4591beda2a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 35377 } } }
I20260929 02:10:57.445001 31351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:57.445576 31350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53f69e57b3954aebaabc7b4591beda2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f69e57b3954aebaabc7b4591beda2a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 35377 } } }
I20260929 02:10:57.446080 31350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a [sys.catalog]: This master's current role is: LEADER
I20260929 02:10:57.448220 31356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:10:57.452301 31356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:10:57.459959 31356 catalog_manager.cc:1347] Generated new cluster ID: 078c706efbff444281c593c40df7b711
I20260929 02:10:57.460072 31356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:10:57.475634 31356 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:10:57.477399 31356 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:10:57.498509 31356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a: Generated new TSK 0
I20260929 02:10:57.499466 31356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.182.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
Entry for principal kudu/127.11.182.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab.
I20260929 02:10:57.562178 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:35377
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:57.891314 31370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:57.891676 31370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:57.891862 31370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:57.899283 31370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:57.899452 31370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:57.899652 31370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:10:57.909561 31370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.193.keytab
--principal=kudu/127.11.182.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:35377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:57.910929 31370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:57.912228 31370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:57.925101 31375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:57.925136 31378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:57.925184 31376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:57.926017 31370 server_base.cc:1034] running on GCE node
I20260929 02:10:57.926635 31370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:57.927632 31370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:57.928869 31370 hybrid_clock.cc:648] HybridClock initialized: now 1790647857928814 us; error 66 us; skew 500 ppm
I20260929 02:10:57.929159 31370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:57 dist-test-slave-q06p krb5kdc[31264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647857, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:57.932327 31370 init.cc:377] Logged in from keytab as kudu/127.11.182.193@KRBTEST.COM (short username kudu)
I20260929 02:10:57.933758 31370 webserver.cc:460] Webserver started at http://127.11.182.193:44999/ using document root <none> and password file <none>
I20260929 02:10:57.934451 31370 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:57.934585 31370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:57.934964 31370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:57.938793 31370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "2b8b86f213774a208bd9880000afc50a"
format_stamp: "Formatted at 2026-09-29 02:10:57 on dist-test-slave-q06p"
server_key: "0e7a47414cb290cfd73388bc07e7fe1c"
server_key_iv: "932c362fa012189c52afacdc2984fc83"
server_key_version: "encryptionkey@0"
I20260929 02:10:57.939582 31370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "2b8b86f213774a208bd9880000afc50a"
format_stamp: "Formatted at 2026-09-29 02:10:57 on dist-test-slave-q06p"
server_key: "0e7a47414cb290cfd73388bc07e7fe1c"
server_key_iv: "932c362fa012189c52afacdc2984fc83"
server_key_version: "encryptionkey@0"
I20260929 02:10:57.944657 31370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:10:57.948076 31385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:57.949303 31370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:10:57.949468 31370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "2b8b86f213774a208bd9880000afc50a"
format_stamp: "Formatted at 2026-09-29 02:10:57 on dist-test-slave-q06p"
server_key: "0e7a47414cb290cfd73388bc07e7fe1c"
server_key_iv: "932c362fa012189c52afacdc2984fc83"
server_key_version: "encryptionkey@0"
I20260929 02:10:57.949652 31370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:57.965162 31370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:57.969365 31370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:57.969667 31370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:57.970688 31370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:57.972040 31370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:57.972118 31370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:57.972294 31370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:57.972347 31370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:58.005334 31370 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:41837
I20260929 02:10:58.005381 31497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:41837 every 8 connection(s)
I20260929 02:10:58.006718 31370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:10:58.009141 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 31370
I20260929 02:10:58.009321 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:10:58.009879 11995 external_mini_cluster.cc:1428] Setting key 24506d6b6698bae5fd19a2962dcdd436
Sep 29 02:10:58 dist-test-slave-q06p krb5kdc[31264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647857, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.193@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:58.031137 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:58.009420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:33801 (local address 127.11.182.254:35377)
0929 02:10:58.009922 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:58.009930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:58.011413 (+ 1483us) server_negotiation.cc:407] Connection header received
0929 02:10:58.014121 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:58.014125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:58.014191 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:58.014285 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:58.016425 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:58.017218 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:58.019753 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:58.020139 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:58.020241 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:58.024039 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:58.024048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:58.024055 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:58.024099 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:58.027818 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:58.028531 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:58.028540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:58.028546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:58.028706 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:58.029119 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:58.029127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:58.029133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:58.029535 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:58.029704 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:58.030367 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 02:10:58.030629 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":248,"threads_started":1}
I20260929 02:10:58.031788 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:58.010055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:35377 (local address 127.11.182.193:33801)
0929 02:10:58.011254 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:58.011301 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:58.013785 (+ 2484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:58.014563 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:58.014578 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:58.015435 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:58.016221 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:58.016257 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:58.017404 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:58.017412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:58.019607 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:58.019625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:58.020378 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:58.020387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:58.020726 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:58.021546 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:58.021568 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:58.023802 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:58.027962 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:58.027969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:58.027982 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:58.028393 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:58.028817 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:58.028820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:58.028822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:58.029002 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:58.029663 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:58.029668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:58.029981 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 02:10:58.030212 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 02:10:58.030583 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":706,"threads_started":1}
I20260929 02:10:58.033456 31498 heartbeater.cc:347] Connected to a master server at 127.11.182.254:35377
I20260929 02:10:58.033766 31498 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:58.034528 31498 heartbeater.cc:511] Master 127.11.182.254:35377 requested a full tablet report, sending...
I20260929 02:10:58.036836 31309 ts_manager.cc:194] Registered new tserver with Master: 2b8b86f213774a208bd9880000afc50a (127.11.182.193:41837)
WARNING: no policy specified for kudu/127.11.182.194@KRBTEST.COM; defaulting to no policy
I20260929 02:10:58.038372 31309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.193@KRBTEST.COM'} at 127.11.182.193:33801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
Entry for principal kudu/127.11.182.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab.
I20260929 02:10:58.070505 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:35377
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:58.395350 31504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:58.395606 31504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:58.395697 31504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:58.402652 31504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:58.402791 31504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:58.403031 31504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:10:58.412559 31504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.194.keytab
--principal=kudu/127.11.182.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:35377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:58.413892 31504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:58.415248 31504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:58.428733 31510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:58.428828 31512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:58.428789 31509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:58.429718 31504 server_base.cc:1034] running on GCE node
I20260929 02:10:58.430228 31504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:58.431138 31504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:58.432329 31504 hybrid_clock.cc:648] HybridClock initialized: now 1790647858432295 us; error 43 us; skew 500 ppm
I20260929 02:10:58.432641 31504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:58 dist-test-slave-q06p krb5kdc[31264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647858, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:58.435662 31504 init.cc:377] Logged in from keytab as kudu/127.11.182.194@KRBTEST.COM (short username kudu)
I20260929 02:10:58.437039 31504 webserver.cc:460] Webserver started at http://127.11.182.194:41391/ using document root <none> and password file <none>
I20260929 02:10:58.437644 31504 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:58.437741 31504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:58.437988 31504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:58.440873 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "7e8c2147e52f4fc2ae04a20253b977f7"
format_stamp: "Formatted at 2026-09-29 02:10:58 on dist-test-slave-q06p"
server_key: "78860a0c7ded65324b66fefb5a1258d4"
server_key_iv: "0defc11ed6880e7dbf1a12c89fe0ab56"
server_key_version: "encryptionkey@0"
I20260929 02:10:58.441614 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "7e8c2147e52f4fc2ae04a20253b977f7"
format_stamp: "Formatted at 2026-09-29 02:10:58 on dist-test-slave-q06p"
server_key: "78860a0c7ded65324b66fefb5a1258d4"
server_key_iv: "0defc11ed6880e7dbf1a12c89fe0ab56"
server_key_version: "encryptionkey@0"
I20260929 02:10:58.446676 31504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:10:58.450290 31519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:58.451838 31504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:10:58.451972 31504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "7e8c2147e52f4fc2ae04a20253b977f7"
format_stamp: "Formatted at 2026-09-29 02:10:58 on dist-test-slave-q06p"
server_key: "78860a0c7ded65324b66fefb5a1258d4"
server_key_iv: "0defc11ed6880e7dbf1a12c89fe0ab56"
server_key_version: "encryptionkey@0"
I20260929 02:10:58.452150 31504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:58.468137 31504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:58.471804 31504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:58.472029 31504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:58.472872 31504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:58.474109 31504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:58.474186 31504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:58.474268 31504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:58.474311 31504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:58.508185 31504 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:37339
I20260929 02:10:58.508232 31631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:37339 every 8 connection(s)
I20260929 02:10:58.509447 31504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:10:58.515012 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 31504
I20260929 02:10:58.515211 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:10:58.515724 11995 external_mini_cluster.cc:1428] Setting key 52ac202657c74f18614cd4d1703872fe
Sep 29 02:10:58 dist-test-slave-q06p krb5kdc[31264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647858, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.194@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:58.532162 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:58.511975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:55863 (local address 127.11.182.254:35377)
0929 02:10:58.512158 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:58.512162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:58.513207 (+ 1045us) server_negotiation.cc:407] Connection header received
0929 02:10:58.515306 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:58.515313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:58.515406 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:58.515491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:58.517303 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:58.518183 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:58.520771 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:58.521162 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:58.521259 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:58.525515 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:58.525520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:58.525523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:58.525553 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:58.528538 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:58.529465 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:58.529473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:58.529480 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:58.529644 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:58.530066 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:58.530074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:58.530080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:58.530477 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:58.530649 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:58.531405 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 02:10:58.531664 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:10:58.532784 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:58.512355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:35377 (local address 127.11.182.194:55863)
0929 02:10:58.513046 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:58.513082 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:58.514972 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:58.515775 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:58.515787 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:58.516463 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:58.517141 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:58.517158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:58.518364 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:58.518369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:58.520615 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:58.520632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:58.521380 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:58.521389 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:58.521717 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:58.522409 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:58.522440 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:58.525293 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:58.528686 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:58.528693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:58.528705 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:58.529324 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:58.529754 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:58.529758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:58.529760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:58.529945 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:58.530620 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:58.530626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:58.531023 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 02:10:58.531257 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 02:10:58.531666 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":280,"threads_started":1}
I20260929 02:10:58.534343 31632 heartbeater.cc:347] Connected to a master server at 127.11.182.254:35377
I20260929 02:10:58.534669 31632 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:58.535444 31632 heartbeater.cc:511] Master 127.11.182.254:35377 requested a full tablet report, sending...
I20260929 02:10:58.536901 31309 ts_manager.cc:194] Registered new tserver with Master: 7e8c2147e52f4fc2ae04a20253b977f7 (127.11.182.194:37339)
I20260929 02:10:58.537731 31309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.194@KRBTEST.COM'} at 127.11.182.194:55863
WARNING: no policy specified for kudu/127.11.182.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
Entry for principal kudu/127.11.182.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab.
I20260929 02:10:58.575336 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:35377
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:10:58.898550 31637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:10:58.898861 31637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:10:58.899044 31637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:10:58.906469 31637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:10:58.906625 31637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:10:58.906822 31637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:10:58.916762 31637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.195.keytab
--principal=kudu/127.11.182.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:35377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:10:58.918080 31637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:10:58.919395 31637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:10:58.932428 31642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:58.932349 31643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:10:58.932528 31645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:10:58.933808 31637 server_base.cc:1034] running on GCE node
I20260929 02:10:58.934520 31637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:10:58.935549 31637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:10:58.936782 31637 hybrid_clock.cc:648] HybridClock initialized: now 1790647858936737 us; error 43 us; skew 500 ppm
I20260929 02:10:58.937090 31637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:10:58 dist-test-slave-q06p krb5kdc[31264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647858, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:10:58.940217 31637 init.cc:377] Logged in from keytab as kudu/127.11.182.195@KRBTEST.COM (short username kudu)
I20260929 02:10:58.941876 31637 webserver.cc:460] Webserver started at http://127.11.182.195:40179/ using document root <none> and password file <none>
I20260929 02:10:58.942540 31637 fs_manager.cc:359] Metadata directory not provided
I20260929 02:10:58.942680 31637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:10:58.943068 31637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:10:58.945871 31637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "0389a65272644e7ba543f6b8314657b4"
format_stamp: "Formatted at 2026-09-29 02:10:58 on dist-test-slave-q06p"
server_key: "8d512b23e6f5128b3551c12f57941983"
server_key_iv: "830b1cdb91cf7c613ff69211288aafa5"
server_key_version: "encryptionkey@0"
I20260929 02:10:58.946653 31637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "0389a65272644e7ba543f6b8314657b4"
format_stamp: "Formatted at 2026-09-29 02:10:58 on dist-test-slave-q06p"
server_key: "8d512b23e6f5128b3551c12f57941983"
server_key_iv: "830b1cdb91cf7c613ff69211288aafa5"
server_key_version: "encryptionkey@0"
I20260929 02:10:58.951774 31637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:10:58.955363 31652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:58.956728 31637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:10:58.956902 31637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "0389a65272644e7ba543f6b8314657b4"
format_stamp: "Formatted at 2026-09-29 02:10:58 on dist-test-slave-q06p"
server_key: "8d512b23e6f5128b3551c12f57941983"
server_key_iv: "830b1cdb91cf7c613ff69211288aafa5"
server_key_version: "encryptionkey@0"
I20260929 02:10:58.957101 31637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:10:58.992019 31637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:10:58.997427 31637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:10:58.997720 31637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:10:58.998755 31637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:10:59.000114 31637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:10:59.000190 31637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:59.000288 31637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:10:59.000330 31637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:10:59.033999 31637 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:44249
I20260929 02:10:59.034087 31764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:44249 every 8 connection(s)
I20260929 02:10:59.035377 31637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:10:59.039877 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 31637
I20260929 02:10:59.040059 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:10:59.040612 11995 external_mini_cluster.cc:1428] Setting key a77b0109ccdf38a11f7beb057dbe33a9
I20260929 02:10:59.041468 31498 heartbeater.cc:503] Master 127.11.182.254:35377 was elected leader, sending a full tablet report...
Sep 29 02:10:59 dist-test-slave-q06p krb5kdc[31264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647858, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.195@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:10:59.057202 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.037980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:49185 (local address 127.11.182.254:35377)
0929 02:10:59.038448 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.038455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.039488 (+ 1033us) server_negotiation.cc:407] Connection header received
0929 02:10:59.041947 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.041952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.042044 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.042163 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:59.044613 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.045416 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.046943 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.047478 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.047584 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.051072 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:59.051077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:59.051080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:59.051113 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:59.053768 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.054624 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.054635 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.054640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.054794 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.055275 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.055284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.055286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.055632 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:59.055786 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.056547 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.056780 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":222,"threads_started":1}
I20260929 02:10:59.058135 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.038603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:35377 (local address 127.11.182.195:49185)
0929 02:10:59.039311 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:59.039350 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:59.041631 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:59.043188 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:59.043200 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:59.043898 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:10:59.044468 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.044486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.045539 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.045542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.046773 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.046784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.047670 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.047675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.047866 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.048468 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:10:59.048542 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:10:59.050793 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:10:59.053921 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:59.053930 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:59.053948 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:59.054488 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:59.054938 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:10:59.054943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:10:59.054947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:10:59.055135 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:10:59.055755 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:10:59.055764 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:10:59.056187 (+ 423us) client_negotiation.cc:769] Sending connection context
0929 02:10:59.056454 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 02:10:59.057012 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":306,"threads_started":1}
I20260929 02:10:59.059522 31765 heartbeater.cc:347] Connected to a master server at 127.11.182.254:35377
I20260929 02:10:59.059831 31765 heartbeater.cc:464] Registering TS with master...
I20260929 02:10:59.060546 31765 heartbeater.cc:511] Master 127.11.182.254:35377 requested a full tablet report, sending...
I20260929 02:10:59.061978 31308 ts_manager.cc:194] Registered new tserver with Master: 0389a65272644e7ba543f6b8314657b4 (127.11.182.195:44249)
I20260929 02:10:59.062924 31308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.182.195@KRBTEST.COM'} at 127.11.182.195:49185
I20260929 02:10:59.068212 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:10:59.084424 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.072017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.11.182.254:35377)
0929 02:10:59.072211 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.072220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.072584 (+ 364us) server_negotiation.cc:407] Connection header received
0929 02:10:59.072778 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.072784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.072855 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.072957 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:59.074310 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.075100 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.076353 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.076782 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.076911 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.079385 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:59.079390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:59.079393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:59.079421 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:59.081748 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.082413 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.082419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.082422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.082552 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.082950 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.082957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.082962 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.083287 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:59.083552 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.083883 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.084108 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:10:59.088217 31308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:10:59.095129 31308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:10:59.135068 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.107004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:41837 (local address 127.0.0.1:40690)
0929 02:10:59.107995 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:59.108036 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:59.108213 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:59.127358 (+ 19145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:59.127366 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:59.127397 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:59.127867 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.127881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.129721 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.129725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.132162 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.132175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.134000 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.134006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.134186 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.134237 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 02:10:59.134378 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 02:10:59.134585 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":49,"thread_start_us":1152,"threads_started":1}
I20260929 02:10:59.135295 31779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.107622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:44249 (local address 127.0.0.1:55372)
0929 02:10:59.126448 (+ 18826us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:59.126471 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:59.126588 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:59.127588 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:59.127594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:59.127637 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:59.128264 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.128278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.130196 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.130201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.132531 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.132546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.134343 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.134349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.134530 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.134553 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:10:59.134659 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:10:59.134796 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18655,"mutex_wait_us":87,"spinlock_wait_cycles":6912,"thread_start_us":17240,"threads_started":1}
I20260929 02:10:59.135618 31783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.108251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.11.182.195:44249)
0929 02:10:59.124454 (+ 16203us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.124469 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.126716 (+ 2247us) server_negotiation.cc:407] Connection header received
0929 02:10:59.126832 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.126861 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.127135 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.127334 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:59.128487 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.129993 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.132687 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.134085 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.134273 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.134451 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.134708 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.134890 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15937,"thread_start_us":16231,"threads_started":1}
I20260929 02:10:59.135999 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.107204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.11.182.193:41837)
0929 02:10:59.126791 (+ 19587us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.126800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.126831 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:10:59.126978 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.126983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.127150 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.127343 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:59.128043 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.129591 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.132337 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.133785 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.133994 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.134189 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.135342 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.135672 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19431,"thread_start_us":19262,"threads_started":1}
I20260929 02:10:59.136492 31781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.107076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:37339 (local address 127.0.0.1:49910)
0929 02:10:59.125213 (+ 18137us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:59.125239 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:59.125396 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:59.127588 (+ 2192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:59.127594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:59.127630 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:59.128279 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.128288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.131629 (+ 3341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.131635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.133379 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.133390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.135651 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.135657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.135838 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.135872 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:10:59.135947 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 02:10:59.136092 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17963,"mutex_wait_us":3,"spinlock_wait_cycles":1280,"thread_start_us":19031,"threads_started":1}
I20260929 02:10:59.136890 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.107748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.11.182.194:37339)
0929 02:10:59.126776 (+ 19028us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.126785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.126809 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:10:59.126908 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.126914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.127130 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.127334 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:59.129786 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.131416 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.133528 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.134942 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.135132 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.136200 (+ 1068us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.136264 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.136411 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18029,"thread_start_us":17916,"threads_started":1}
I20260929 02:10:59.139791 31700 tablet_service.cc:1467] Processing CreateTablet for tablet 4572dcead06b4616b73e1c992df94e2a (DEFAULT_TABLE table=test-table [id=50df97154c7c4773a8810544376a6438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:59.139791 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 4572dcead06b4616b73e1c992df94e2a (DEFAULT_TABLE table=test-table [id=50df97154c7c4773a8810544376a6438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:59.140283 31567 tablet_service.cc:1467] Processing CreateTablet for tablet 4572dcead06b4616b73e1c992df94e2a (DEFAULT_TABLE table=test-table [id=50df97154c7c4773a8810544376a6438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:10:59.141302 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4572dcead06b4616b73e1c992df94e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:59.141302 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4572dcead06b4616b73e1c992df94e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:59.141490 31567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4572dcead06b4616b73e1c992df94e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:10:59.152112 31784 tablet_bootstrap.cc:492] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: Bootstrap starting.
I20260929 02:10:59.155325 31784 tablet_bootstrap.cc:654] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:59.155380 31785 tablet_bootstrap.cc:492] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: Bootstrap starting.
I20260929 02:10:59.156286 31786 tablet_bootstrap.cc:492] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: Bootstrap starting.
I20260929 02:10:59.157280 31784 log.cc:824] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:59.159421 31786 tablet_bootstrap.cc:654] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:59.159584 31785 tablet_bootstrap.cc:654] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: Neither blocks nor log segments found. Creating new log.
I20260929 02:10:59.160823 31786 log.cc:824] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:59.160838 31785 log.cc:824] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:10:59.161885 31784 tablet_bootstrap.cc:492] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: No bootstrap required, opened a new log
I20260929 02:10:59.162149 31784 ts_tablet_manager.cc:1397] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 02:10:59.163841 31785 tablet_bootstrap.cc:492] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: No bootstrap required, opened a new log
I20260929 02:10:59.164009 31786 tablet_bootstrap.cc:492] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: No bootstrap required, opened a new log
I20260929 02:10:59.164105 31785 ts_tablet_manager.cc:1397] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:10:59.164243 31786 ts_tablet_manager.cc:1397] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:10:59.171787 31785 raft_consensus.cc:348] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } }
I20260929 02:10:59.172199 31785 raft_consensus.cc:374] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:59.172307 31785 raft_consensus.cc:729] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b8b86f213774a208bd9880000afc50a, State: Initialized, Role: FOLLOWER
I20260929 02:10:59.172727 31784 raft_consensus.cc:348] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } }
I20260929 02:10:59.172961 31785 consensus_queue.cc:260] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } }
I20260929 02:10:59.173162 31784 raft_consensus.cc:374] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:59.173269 31784 raft_consensus.cc:729] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8c2147e52f4fc2ae04a20253b977f7, State: Initialized, Role: FOLLOWER
I20260929 02:10:59.173946 31784 consensus_queue.cc:260] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } }
I20260929 02:10:59.174831 31785 ts_tablet_manager.cc:1428] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 02:10:59.174906 31786 raft_consensus.cc:348] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } }
I20260929 02:10:59.175385 31786 raft_consensus.cc:374] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:10:59.175504 31786 raft_consensus.cc:729] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0389a65272644e7ba543f6b8314657b4, State: Initialized, Role: FOLLOWER
I20260929 02:10:59.176198 31786 consensus_queue.cc:260] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } }
I20260929 02:10:59.177428 31765 heartbeater.cc:503] Master 127.11.182.254:35377 was elected leader, sending a full tablet report...
I20260929 02:10:59.178237 31786 ts_tablet_manager.cc:1428] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 02:10:59.181017 31632 heartbeater.cc:503] Master 127.11.182.254:35377 was elected leader, sending a full tablet report...
I20260929 02:10:59.181694 31784 ts_tablet_manager.cc:1428] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: Time spent starting tablet: real 0.019s user 0.011s sys 0.008s
W20260929 02:10:59.259173 31499 tablet.cc:2369] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:10:59.261392 31633 tablet.cc:2369] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:10:59.287165 31766 tablet.cc:2369] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:10:59.444653 31791 raft_consensus.cc:482] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:10:59.445000 31791 raft_consensus.cc:504] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } }
I20260929 02:10:59.446790 31791 leader_election.cc:288] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b8b86f213774a208bd9880000afc50a (127.11.182.193:41837), 0389a65272644e7ba543f6b8314657b4 (127.11.182.195:44249)
I20260929 02:10:59.456313 31783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.447344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:41999 (local address 127.11.182.195:44249)
0929 02:10:59.447671 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.447677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.448283 (+ 606us) server_negotiation.cc:407] Connection header received
0929 02:10:59.448493 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.448499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.448596 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.448696 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:59.449484 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.450606 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.453775 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.455072 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.455250 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.455319 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.455907 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.456054 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260929 02:10:59.456313 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.447320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:46621 (local address 127.11.182.193:41837)
0929 02:10:59.447538 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.447545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.448289 (+ 744us) server_negotiation.cc:407] Connection header received
0929 02:10:59.448494 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.448499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.448596 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.448696 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:10:59.449484 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.450606 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.453816 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.455086 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.455242 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.455319 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.455926 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.456056 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:10:59.456559 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.447206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:41837 (local address 127.11.182.194:46621)
0929 02:10:59.448174 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:59.448189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:59.448344 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:59.448733 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:59.448738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:59.448790 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:59.449311 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.449332 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.450770 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.450775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.453596 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.453618 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.455373 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.455378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.455648 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.455760 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 02:10:59.455857 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 02:10:59.456054 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":292,"spinlock_wait_cycles":29952,"thread_start_us":490,"threads_started":1}
I20260929 02:10:59.456919 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.447101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:44249 (local address 127.11.182.194:41999)
0929 02:10:59.448144 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0929 02:10:59.448169 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:10:59.448362 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:10:59.448733 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:10:59.448738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:10:59.448769 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:10:59.449336 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.449345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.450769 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.450775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.453615 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:10:59.453625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.455371 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.455378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:10:59.455683 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.455763 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 02:10:59.455860 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 02:10:59.456054 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":303,"spinlock_wait_cycles":49408,"thread_start_us":517,"threads_started":1}
I20260929 02:10:59.457661 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4572dcead06b4616b73e1c992df94e2a" candidate_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b8b86f213774a208bd9880000afc50a" is_pre_election: true
I20260929 02:10:59.457914 31720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4572dcead06b4616b73e1c992df94e2a" candidate_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0389a65272644e7ba543f6b8314657b4" is_pre_election: true
I20260929 02:10:59.458225 31453 raft_consensus.cc:2454] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e8c2147e52f4fc2ae04a20253b977f7 in term 0.
I20260929 02:10:59.458431 31720 raft_consensus.cc:2454] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e8c2147e52f4fc2ae04a20253b977f7 in term 0.
I20260929 02:10:59.459029 31521 leader_election.cc:302] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0389a65272644e7ba543f6b8314657b4, 2b8b86f213774a208bd9880000afc50a, 7e8c2147e52f4fc2ae04a20253b977f7; no voters:
I20260929 02:10:59.459436 31791 raft_consensus.cc:2781] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:10:59.459591 31791 raft_consensus.cc:482] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:10:59.459710 31791 raft_consensus.cc:3037] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:59.461990 31791 raft_consensus.cc:504] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } }
I20260929 02:10:59.462545 31791 leader_election.cc:288] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [CANDIDATE]: Term 1 election: Requested vote from peers 2b8b86f213774a208bd9880000afc50a (127.11.182.193:41837), 0389a65272644e7ba543f6b8314657b4 (127.11.182.195:44249)
I20260929 02:10:59.463049 31720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4572dcead06b4616b73e1c992df94e2a" candidate_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0389a65272644e7ba543f6b8314657b4"
I20260929 02:10:59.463049 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4572dcead06b4616b73e1c992df94e2a" candidate_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b8b86f213774a208bd9880000afc50a"
I20260929 02:10:59.463258 31720 raft_consensus.cc:3037] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:59.463238 31453 raft_consensus.cc:3037] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:10:59.466337 31720 raft_consensus.cc:2454] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8c2147e52f4fc2ae04a20253b977f7 in term 1.
I20260929 02:10:59.466337 31453 raft_consensus.cc:2454] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8c2147e52f4fc2ae04a20253b977f7 in term 1.
I20260929 02:10:59.466718 31521 leader_election.cc:302] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0389a65272644e7ba543f6b8314657b4, 2b8b86f213774a208bd9880000afc50a, 7e8c2147e52f4fc2ae04a20253b977f7; no voters:
I20260929 02:10:59.467013 31791 raft_consensus.cc:2781] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:10:59.467394 31791 raft_consensus.cc:686] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 1 LEADER]: Becoming Leader. State: Replica: 7e8c2147e52f4fc2ae04a20253b977f7, State: Running, Role: LEADER
I20260929 02:10:59.467847 31791 consensus_queue.cc:237] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } }
I20260929 02:10:59.472597 31308 catalog_manager.cc:5495] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e8c2147e52f4fc2ae04a20253b977f7 (127.11.182.194). New cstate: current_term: 1 leader_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e8c2147e52f4fc2ae04a20253b977f7" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 37339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 } health_report { overall_health: UNKNOWN } } }
I20260929 02:10:59.572577 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.567094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.11.182.194:37339)
0929 02:10:59.567340 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.567346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.567366 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:10:59.567583 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.567587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.567677 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.567825 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:10:59.568209 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.568922 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.570340 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.570767 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.570931 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.571433 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:10:59.571949 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:10:59.572123 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.572184 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.572282 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 02:10:59.581308 31453 raft_consensus.cc:1261] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [term 1 FOLLOWER]: Refusing update from remote peer 7e8c2147e52f4fc2ae04a20253b977f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:59.581306 31720 raft_consensus.cc:1261] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [term 1 FOLLOWER]: Refusing update from remote peer 7e8c2147e52f4fc2ae04a20253b977f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:10:59.582405 31791 consensus_queue.cc:1035] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0389a65272644e7ba543f6b8314657b4" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 44249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:59.582722 31795 consensus_queue.cc:1035] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b8b86f213774a208bd9880000afc50a" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:10:59.602435 31799 mvcc.cc:205] Tried to move back new op lower bound from 7334493632834629632 to 7334493632383819776. Current Snapshot: MvccSnapshot[applied={T|T < 7334493632834629632 or (T in {7334493632834629632})}]
I20260929 02:10:59.614401 31308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:59.614707 31308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:10:59.618741 31308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53f69e57b3954aebaabc7b4591beda2a: Sending DeleteTablet for 3 replicas of tablet 4572dcead06b4616b73e1c992df94e2a
I20260929 02:10:59.620122 31700 tablet_service.cc:1514] Processing DeleteTablet for tablet 4572dcead06b4616b73e1c992df94e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:59 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:55372
I20260929 02:10:59.620118 31567 tablet_service.cc:1514] Processing DeleteTablet for tablet 4572dcead06b4616b73e1c992df94e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:59 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:49910
I20260929 02:10:59.620167 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 4572dcead06b4616b73e1c992df94e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:10:59 UTC) from {username='kudu', principal='kudu/127.11.182.254@KRBTEST.COM'} at 127.0.0.1:40690
I20260929 02:10:59.622316 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 31370
I20260929 02:10:59.639183 31810 tablet_replica.cc:330] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: stopping tablet replica
I20260929 02:10:59.639691 31809 tablet_replica.cc:330] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: stopping tablet replica
I20260929 02:10:59.639652 31810 raft_consensus.cc:2229] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:59.640107 31809 raft_consensus.cc:2229] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:10:59.640228 31810 raft_consensus.cc:2258] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:59.642587 31810 ts_tablet_manager.cc:1905] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:59.640594 31809 raft_consensus.cc:2258] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:59.645174 31809 ts_tablet_manager.cc:1905] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:10:59 dist-test-slave-q06p krb5kdc[31264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.193@KRBTEST.COM
I20260929 02:10:59.648619 31808 tablet_replica.cc:330] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: stopping tablet replica
I20260929 02:10:59.649084 31808 raft_consensus.cc:2229] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:10:59.649899 31808 raft_consensus.cc:2258] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:10:59.650710 31810 ts_tablet_manager.cc:1918] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:59.650926 31810 log.cc:1196] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/4572dcead06b4616b73e1c992df94e2a
I20260929 02:10:59.651871 31810 ts_tablet_manager.cc:1939] T 4572dcead06b4616b73e1c992df94e2a P 2b8b86f213774a208bd9880000afc50a: Deleting consensus metadata
I20260929 02:10:59.653987 31294 catalog_manager.cc:4841] TS 2b8b86f213774a208bd9880000afc50a (127.11.182.193:41837): tablet 4572dcead06b4616b73e1c992df94e2a (table test-table [id=50df97154c7c4773a8810544376a6438]) successfully deleted
I20260929 02:10:59.656219 31809 ts_tablet_manager.cc:1918] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:59.656435 31809 log.cc:1196] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/4572dcead06b4616b73e1c992df94e2a
I20260929 02:10:59.656971 31809 ts_tablet_manager.cc:1939] T 4572dcead06b4616b73e1c992df94e2a P 0389a65272644e7ba543f6b8314657b4: Deleting consensus metadata
I20260929 02:10:59.657800 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.623256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.11.182.193:41837)
0929 02:10:59.623488 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.623497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.629454 (+ 5957us) server_negotiation.cc:407] Connection header received
0929 02:10:59.630482 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.630492 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.630598 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.630691 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:59.634856 (+ 4165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.636358 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.641832 (+ 5474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.642764 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.643120 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.649159 (+ 6039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:59.649176 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:59.649196 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:59.649238 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:59.653732 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.655029 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.655040 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.655044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.655249 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.655708 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.655714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.655717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.656182 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:59.656483 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.657176 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.657414 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:10:59.658725 31493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:59.658886 31295 catalog_manager.cc:4841] TS 0389a65272644e7ba543f6b8314657b4 (127.11.182.195:44249): tablet 4572dcead06b4616b73e1c992df94e2a (table test-table [id=50df97154c7c4773a8810544376a6438]) successfully deleted
I20260929 02:10:59.661037 31808 ts_tablet_manager.cc:1905] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:10:59.669644 31808 ts_tablet_manager.cc:1918] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:10:59.669862 31808 log.cc:1196] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/4572dcead06b4616b73e1c992df94e2a
I20260929 02:10:59.670444 31808 ts_tablet_manager.cc:1939] T 4572dcead06b4616b73e1c992df94e2a P 7e8c2147e52f4fc2ae04a20253b977f7: Deleting consensus metadata
I20260929 02:10:59.672541 31293 catalog_manager.cc:4841] TS 7e8c2147e52f4fc2ae04a20253b977f7 (127.11.182.194:37339): tablet 4572dcead06b4616b73e1c992df94e2a (table test-table [id=50df97154c7c4773a8810544376a6438]) successfully deleted
I20260929 02:10:59.752321 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 31370
I20260929 02:10:59.775756 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 31504
Sep 29 02:10:59 dist-test-slave-q06p krb5kdc[31264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.194@KRBTEST.COM
I20260929 02:10:59.792980 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.776563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.11.182.194:37339)
0929 02:10:59.776750 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.776755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.776816 (+ 61us) server_negotiation.cc:407] Connection header received
0929 02:10:59.776991 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.776995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.777078 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.777159 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:59.778753 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.779647 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.781466 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.781873 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.781963 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.787203 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:59.787222 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:59.787244 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:59.787283 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:59.790270 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.790995 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.791003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.791005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.791130 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.791512 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.791517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.791520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.791863 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:59.792216 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.792404 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.792620 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:10:59.793675 31627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:10:59.906499 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 31504
I20260929 02:10:59.925763 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 31637
Sep 29 02:10:59 dist-test-slave-q06p krb5kdc[31264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.195@KRBTEST.COM
I20260929 02:10:59.941330 31783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:10:59.926490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.11.182.195:44249)
0929 02:10:59.926674 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:10:59.926678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:10:59.926697 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:10:59.926810 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:10:59.926814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:10:59.926933 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:10:59.927028 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:10:59.928344 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.929060 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.930343 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:10:59.930750 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:10:59.930950 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:10:59.935459 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:10:59.935476 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:10:59.935494 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:10:59.935531 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:10:59.938396 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.939231 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.939241 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.939243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.939371 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:10:59.939761 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:10:59.939766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:10:59.939769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:10:59.940142 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:10:59.940454 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 02:10:59.940763 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 02:10:59.940975 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:10:59.941998 31760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:00.031051 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 31637
I20260929 02:11:00.047675 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 31277
I20260929 02:11:00.048746 31339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:00.159071 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 31277
2026-09-29T02:11:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4522 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T02:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:00Z Disabled control of system clock
I20260929 02:11:00.235527 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34091
--webserver_interface=127.11.182.254
--webserver_port=0
--builtin_ntp_servers=127.11.182.212:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:34091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:11:00.559427 31819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:00.559698 31819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:00.559778 31819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:00.566720 31819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:00.566818 31819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:00.566906 31819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:00.566983 31819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:00.567029 31819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:00.576251 31819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:34091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:00.577517 31819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:00.579260 31819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:00.591495 31825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:00.591455 31824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:00.591552 31827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:00.592948 31819 server_base.cc:1034] running on GCE node
I20260929 02:11:00.593698 31819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:00.595055 31819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:00.596276 31819 hybrid_clock.cc:648] HybridClock initialized: now 1790647860596236 us; error 51 us; skew 500 ppm
I20260929 02:11:00.596637 31819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:11:00.598835 31819 webserver.cc:460] Webserver started at http://127.11.182.254:34105/ using document root <none> and password file <none>
I20260929 02:11:00.599587 31819 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:00.599702 31819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:00.600042 31819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:00.602749 31819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "61ede32ff5fb465eaecb24522c7285b7"
format_stamp: "Formatted at 2026-09-29 02:11:00 on dist-test-slave-q06p"
server_key: "d4dceb5a3ec5a08e48296f37120c6cd9"
server_key_iv: "d8322716c3d553635f9810b4a7e83e27"
server_key_version: "encryptionkey@0"
I20260929 02:11:00.603586 31819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "61ede32ff5fb465eaecb24522c7285b7"
format_stamp: "Formatted at 2026-09-29 02:11:00 on dist-test-slave-q06p"
server_key: "d4dceb5a3ec5a08e48296f37120c6cd9"
server_key_iv: "d8322716c3d553635f9810b4a7e83e27"
server_key_version: "encryptionkey@0"
I20260929 02:11:00.608454 31819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:11:00.611722 31833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:00.612922 31819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:11:00.613165 31819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "61ede32ff5fb465eaecb24522c7285b7"
format_stamp: "Formatted at 2026-09-29 02:11:00 on dist-test-slave-q06p"
server_key: "d4dceb5a3ec5a08e48296f37120c6cd9"
server_key_iv: "d8322716c3d553635f9810b4a7e83e27"
server_key_version: "encryptionkey@0"
I20260929 02:11:00.613430 31819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:00.639460 31819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:00.640261 31819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:00.640544 31819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:00.658342 31819 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:34091
I20260929 02:11:00.658349 31884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:34091 every 8 connection(s)
I20260929 02:11:00.659659 31819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:00.663260 31885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:00.668066 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 31819
I20260929 02:11:00.668263 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:00.668819 11995 external_mini_cluster.cc:1428] Setting key fef6c17014ef8aa46203451d382646f3
I20260929 02:11:00.674183 31885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7: Bootstrap starting.
I20260929 02:11:00.677091 31885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:00.678064 31885 log.cc:824] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:00.680898 31885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7: No bootstrap required, opened a new log
I20260929 02:11:00.681348 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:00.670632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.11.182.254:34091)
0929 02:11:00.671676 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:00.671685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:00.671725 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:11:00.673310 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:00.673331 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:00.673729 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:00.674121 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:00.674655 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:00.675965 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:00.677210 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:00.677859 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:00.678042 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:00.679219 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:00.679227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:00.679246 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:00.679761 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:00.679832 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:00.679852 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:00.679982 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:00.680159 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 02:11:00.680447 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":385,"threads_started":1}
I20260929 02:11:00.688455 31885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ede32ff5fb465eaecb24522c7285b7" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34091 } }
I20260929 02:11:00.688798 31885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:00.688905 31885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ede32ff5fb465eaecb24522c7285b7, State: Initialized, Role: FOLLOWER
I20260929 02:11:00.689474 31885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ede32ff5fb465eaecb24522c7285b7" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34091 } }
I20260929 02:11:00.689739 31885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:00.689881 31885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:00.690073 31885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:00.692298 31885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ede32ff5fb465eaecb24522c7285b7" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34091 } }
I20260929 02:11:00.692830 31885 leader_election.cc:302] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61ede32ff5fb465eaecb24522c7285b7; no voters:
I20260929 02:11:00.693439 31885 leader_election.cc:288] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:00.693588 31890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:00.694383 31890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [term 1 LEADER]: Becoming Leader. State: Replica: 61ede32ff5fb465eaecb24522c7285b7, State: Running, Role: LEADER
I20260929 02:11:00.694978 31890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ede32ff5fb465eaecb24522c7285b7" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34091 } }
I20260929 02:11:00.695729 31885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:00.697906 31891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61ede32ff5fb465eaecb24522c7285b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ede32ff5fb465eaecb24522c7285b7" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34091 } } }
I20260929 02:11:00.698393 31891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:00.697901 31892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61ede32ff5fb465eaecb24522c7285b7. Latest consensus state: current_term: 1 leader_uuid: "61ede32ff5fb465eaecb24522c7285b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ede32ff5fb465eaecb24522c7285b7" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34091 } } }
I20260929 02:11:00.698622 31892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:00.700553 31896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:00.706132 31896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:00.714838 31896 catalog_manager.cc:1347] Generated new cluster ID: 02fbb0f02f5f4e01984893f06f769130
I20260929 02:11:00.714967 31896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:00.732089 31896 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:00.733350 31896 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:00.746104 31896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7: Generated new TSK 0
I20260929 02:11:00.747006 31896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:00.756458 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.193:0
--local_ip_for_outbound_sockets=127.11.182.193
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34091
--builtin_ntp_servers=127.11.182.212:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:11:01.085554 31909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:01.085820 31909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:01.085906 31909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:01.093048 31909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:01.093147 31909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:01.093295 31909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.193
I20260929 02:11:01.103636 31909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.182.193
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.193
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:01.104955 31909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:01.106666 31909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:01.119441 31914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:01.119480 31915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:01.119522 31917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:01.120944 31909 server_base.cc:1034] running on GCE node
I20260929 02:11:01.121518 31909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:01.122400 31909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:01.123641 31909 hybrid_clock.cc:648] HybridClock initialized: now 1790647861123611 us; error 41 us; skew 500 ppm
I20260929 02:11:01.123945 31909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:11:01.126016 31909 webserver.cc:460] Webserver started at http://127.11.182.193:32985/ using document root <none> and password file <none>
I20260929 02:11:01.126683 31909 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:01.126816 31909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:01.127142 31909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:01.129825 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/data/instance:
uuid: "fb81501e3e9c400589972733d773294b"
format_stamp: "Formatted at 2026-09-29 02:11:01 on dist-test-slave-q06p"
server_key: "ec02540897166d65071f8114ffdacddd"
server_key_iv: "50d63a8c85f3b8fc4f286ef95ceb4ea1"
server_key_version: "encryptionkey@0"
I20260929 02:11:01.130574 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance:
uuid: "fb81501e3e9c400589972733d773294b"
format_stamp: "Formatted at 2026-09-29 02:11:01 on dist-test-slave-q06p"
server_key: "ec02540897166d65071f8114ffdacddd"
server_key_iv: "50d63a8c85f3b8fc4f286ef95ceb4ea1"
server_key_version: "encryptionkey@0"
I20260929 02:11:01.135457 31909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 02:11:01.138824 31923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:01.140215 31909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:11:01.140419 31909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/wal
uuid: "fb81501e3e9c400589972733d773294b"
format_stamp: "Formatted at 2026-09-29 02:11:01 on dist-test-slave-q06p"
server_key: "ec02540897166d65071f8114ffdacddd"
server_key_iv: "50d63a8c85f3b8fc4f286ef95ceb4ea1"
server_key_version: "encryptionkey@0"
I20260929 02:11:01.140648 31909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:01.157867 31909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:01.158665 31909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:01.159021 31909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:01.159932 31909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:11:01.161309 31909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:11:01.161432 31909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:01.161566 31909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:11:01.161662 31909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:01.194298 31909 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.193:38793
I20260929 02:11:01.194351 32035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.193:38793 every 8 connection(s)
I20260929 02:11:01.195683 31909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/data/info.pb
I20260929 02:11:01.203547 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 31909
I20260929 02:11:01.203720 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/wal/instance
I20260929 02:11:01.204219 11995 external_mini_cluster.cc:1428] Setting key c6287e22bd3c474f2d35ab3ed5f0e7f7
I20260929 02:11:01.207191 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.194:0
--local_ip_for_outbound_sockets=127.11.182.194
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34091
--builtin_ntp_servers=127.11.182.212:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:11:01.209949 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:01.198175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:58849 (local address 127.11.182.254:34091)
0929 02:11:01.198602 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:01.198608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:01.199943 (+ 1335us) server_negotiation.cc:407] Connection header received
0929 02:11:01.201896 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:01.201900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:01.202006 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:01.202097 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:01.203324 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:01.204417 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:01.206583 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:01.207316 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:01.207465 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:01.209213 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:01.209218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:01.209221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:01.209238 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:01.209281 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:01.209293 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:01.209369 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:01.209624 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 02:11:01.209701 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":206,"threads_started":1}
I20260929 02:11:01.211395 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:01.198823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:34091 (local address 127.11.182.193:58849)
0929 02:11:01.199798 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:01.199837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:01.201668 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:01.202298 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:01.202306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:01.202386 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:01.203145 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:01.203181 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:01.204580 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:01.204584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:01.206439 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:01.206452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:01.207742 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:01.207753 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:01.208089 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:01.208871 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:11:01.208877 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:01.208929 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:11:01.209029 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:01.209400 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:01.209413 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:01.209424 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:11:01.209612 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 02:11:01.210021 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":265,"threads_started":1}
I20260929 02:11:01.212970 32036 heartbeater.cc:347] Connected to a master server at 127.11.182.254:34091
I20260929 02:11:01.213284 32036 heartbeater.cc:464] Registering TS with master...
I20260929 02:11:01.214075 32036 heartbeater.cc:511] Master 127.11.182.254:34091 requested a full tablet report, sending...
I20260929 02:11:01.216212 31850 ts_manager.cc:194] Registered new tserver with Master: fb81501e3e9c400589972733d773294b (127.11.182.193:38793)
I20260929 02:11:01.217881 31850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.193:58849
W20260929 02:11:01.529747 32040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:01.530001 32040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:01.530153 32040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:01.537344 32040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:01.537453 32040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:01.537621 32040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.194
I20260929 02:11:01.546808 32040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.182.194
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.194
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:01.548086 32040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:01.549837 32040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:01.562633 32045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:01.562547 32046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:01.562605 32048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:01.563508 32040 server_base.cc:1034] running on GCE node
I20260929 02:11:01.564136 32040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:01.565025 32040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:01.566272 32040 hybrid_clock.cc:648] HybridClock initialized: now 1790647861566233 us; error 54 us; skew 500 ppm
I20260929 02:11:01.566583 32040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:11:01.568745 32040 webserver.cc:460] Webserver started at http://127.11.182.194:37917/ using document root <none> and password file <none>
I20260929 02:11:01.569434 32040 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:01.569566 32040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:01.569877 32040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:01.572765 32040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/data/instance:
uuid: "f36b1baba1b24032bf9a714c05a3550b"
format_stamp: "Formatted at 2026-09-29 02:11:01 on dist-test-slave-q06p"
server_key: "52e0478532d1ec184fe609b271626890"
server_key_iv: "b5a0f080d1175cf7a4c76752536ecb9e"
server_key_version: "encryptionkey@0"
I20260929 02:11:01.573562 32040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance:
uuid: "f36b1baba1b24032bf9a714c05a3550b"
format_stamp: "Formatted at 2026-09-29 02:11:01 on dist-test-slave-q06p"
server_key: "52e0478532d1ec184fe609b271626890"
server_key_iv: "b5a0f080d1175cf7a4c76752536ecb9e"
server_key_version: "encryptionkey@0"
I20260929 02:11:01.578838 32040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:11:01.582322 32054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:01.583712 32040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:11:01.583837 32040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/wal
uuid: "f36b1baba1b24032bf9a714c05a3550b"
format_stamp: "Formatted at 2026-09-29 02:11:01 on dist-test-slave-q06p"
server_key: "52e0478532d1ec184fe609b271626890"
server_key_iv: "b5a0f080d1175cf7a4c76752536ecb9e"
server_key_version: "encryptionkey@0"
I20260929 02:11:01.584007 32040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:01.601128 32040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:01.602097 32040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:01.602434 32040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:01.603629 32040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:11:01.605118 32040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:11:01.605239 32040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:01.605340 32040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:11:01.605436 32040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:01.638183 32040 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.194:43063
I20260929 02:11:01.638196 32166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.194:43063 every 8 connection(s)
I20260929 02:11:01.639616 32040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/data/info.pb
I20260929 02:11:01.643924 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 32040
I20260929 02:11:01.644162 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/wal/instance
I20260929 02:11:01.644773 11995 external_mini_cluster.cc:1428] Setting key 78ca6daf18fbc63265cc23985b4842ba
I20260929 02:11:01.647863 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.182.195:0
--local_ip_for_outbound_sockets=127.11.182.195
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34091
--builtin_ntp_servers=127.11.182.212:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:11:01.653564 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:01.642090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.194:52601 (local address 127.11.182.254:34091)
0929 02:11:01.642255 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:01.642260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:01.643659 (+ 1399us) server_negotiation.cc:407] Connection header received
0929 02:11:01.645964 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:01.645971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:01.646059 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:01.646184 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:01.647299 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:01.648120 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:01.650598 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:01.651212 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:01.651350 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:01.652825 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:01.652830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:01.652833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:01.652852 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:01.652889 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:01.652901 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:01.652980 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:01.653228 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 02:11:01.653306 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:11:01.654999 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:01.642620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:34091 (local address 127.11.182.194:52601)
0929 02:11:01.643485 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:01.643526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:01.645713 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:01.646353 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:01.646360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:01.646441 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:01.647105 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:01.647131 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:01.648276 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:01.648283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:01.650437 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:01.650454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:01.651433 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:01.651442 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:01.651780 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:01.652486 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:11:01.652492 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:01.652545 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:11:01.652637 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:01.653003 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:01.653015 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:01.653024 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:11:01.653200 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 02:11:01.653613 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":295,"threads_started":1}
I20260929 02:11:01.656581 32167 heartbeater.cc:347] Connected to a master server at 127.11.182.254:34091
I20260929 02:11:01.656914 32167 heartbeater.cc:464] Registering TS with master...
I20260929 02:11:01.657835 32167 heartbeater.cc:511] Master 127.11.182.254:34091 requested a full tablet report, sending...
I20260929 02:11:01.659487 31850 ts_manager.cc:194] Registered new tserver with Master: f36b1baba1b24032bf9a714c05a3550b (127.11.182.194:43063)
I20260929 02:11:01.660332 31850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.194:52601
W20260929 02:11:01.982136 32170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:01.982415 32170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:01.982503 32170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:01.989979 32170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:01.990074 32170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:01.990266 32170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.182.195
I20260929 02:11:01.999400 32170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.182.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.182.195
--webserver_port=0
--tserver_master_addrs=127.11.182.254:34091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.182.195
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:02.000751 32170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:02.002566 32170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:02.015604 32178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:02.015703 32175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:02.015388 32176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:02.016270 32170 server_base.cc:1034] running on GCE node
I20260929 02:11:02.017011 32170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:02.018082 32170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:02.019308 32170 hybrid_clock.cc:648] HybridClock initialized: now 1790647862019259 us; error 42 us; skew 500 ppm
I20260929 02:11:02.019632 32170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:11:02.021847 32170 webserver.cc:460] Webserver started at http://127.11.182.195:32987/ using document root <none> and password file <none>
I20260929 02:11:02.022543 32170 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:02.022675 32170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:02.023043 32170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:02.026012 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/data/instance:
uuid: "de122bec0d0e4f7a82081f24c8efed24"
format_stamp: "Formatted at 2026-09-29 02:11:02 on dist-test-slave-q06p"
server_key: "6d99d37d6f5b2ddb3fae855ec9b1409e"
server_key_iv: "d5ec0cdc27940410b1f7a28f0d46d4d3"
server_key_version: "encryptionkey@0"
I20260929 02:11:02.026824 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance:
uuid: "de122bec0d0e4f7a82081f24c8efed24"
format_stamp: "Formatted at 2026-09-29 02:11:02 on dist-test-slave-q06p"
server_key: "6d99d37d6f5b2ddb3fae855ec9b1409e"
server_key_iv: "d5ec0cdc27940410b1f7a28f0d46d4d3"
server_key_version: "encryptionkey@0"
I20260929 02:11:02.031962 32170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 02:11:02.035351 32184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:02.036608 32170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:11:02.036784 32170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/wal
uuid: "de122bec0d0e4f7a82081f24c8efed24"
format_stamp: "Formatted at 2026-09-29 02:11:02 on dist-test-slave-q06p"
server_key: "6d99d37d6f5b2ddb3fae855ec9b1409e"
server_key_iv: "d5ec0cdc27940410b1f7a28f0d46d4d3"
server_key_version: "encryptionkey@0"
I20260929 02:11:02.036998 32170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:02.058339 32170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:02.059322 32170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:02.059664 32170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:02.060662 32170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:11:02.062150 32170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:11:02.062279 32170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:02.062415 32170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:11:02.062503 32170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:02.095784 32170 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.195:45847
I20260929 02:11:02.095834 32296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.195:45847 every 8 connection(s)
I20260929 02:11:02.097246 32170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/data/info.pb
I20260929 02:11:02.099257 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 32170
I20260929 02:11:02.099469 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/wal/instance
I20260929 02:11:02.100009 11995 external_mini_cluster.cc:1428] Setting key 47b3f957457107f11584af74e39b6ab4
I20260929 02:11:02.109970 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.100245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.195:38227 (local address 127.11.182.254:34091)
0929 02:11:02.100462 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.100468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.102190 (+ 1722us) server_negotiation.cc:407] Connection header received
0929 02:11:02.104087 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.104099 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.104166 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.104245 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:02.105235 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.105965 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.107524 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.107984 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.108104 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.109296 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:02.109301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:02.109304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:02.109322 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:02.109353 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:02.109363 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:02.109433 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.109623 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.109735 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:11:02.110972 32299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.100841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:34091 (local address 127.11.182.195:38227)
0929 02:11:02.102030 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.102074 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:02.103856 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:02.104447 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:02.104455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:02.104539 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:02.105094 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.105110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.106088 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.106092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.107381 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.107393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.108168 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.108172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.108371 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.109005 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:11:02.109007 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:02.109051 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:11:02.109129 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:02.109447 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:02.109454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:02.109460 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:11:02.109595 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 02:11:02.109878 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":393,"threads_started":1}
I20260929 02:11:02.112413 32297 heartbeater.cc:347] Connected to a master server at 127.11.182.254:34091
I20260929 02:11:02.112690 32297 heartbeater.cc:464] Registering TS with master...
I20260929 02:11:02.113509 32297 heartbeater.cc:511] Master 127.11.182.254:34091 requested a full tablet report, sending...
I20260929 02:11:02.115193 31850 ts_manager.cc:194] Registered new tserver with Master: de122bec0d0e4f7a82081f24c8efed24 (127.11.182.195:45847)
I20260929 02:11:02.116124 31850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.182.195:38227
I20260929 02:11:02.127902 11995 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:11:02.137778 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.131853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.11.182.254:34091)
0929 02:11:02.132047 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.132053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.132463 (+ 410us) server_negotiation.cc:407] Connection header received
0929 02:11:02.132661 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.132666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.132744 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.132848 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:02.133342 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.134030 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.135396 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.135855 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.135987 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.137067 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:02.137072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:02.137075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:02.137092 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:02.137124 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:02.137136 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:02.137231 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.137380 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.137479 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:11:02.141866 31850 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:11:02.148967 31850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:11:02.171751 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.161814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.11.182.194:43063)
0929 02:11:02.163161 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.163169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.163504 (+ 335us) server_negotiation.cc:407] Connection header received
0929 02:11:02.163606 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.163615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.163889 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.164072 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:11:02.165083 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.166526 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.169159 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.170641 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.170873 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.171126 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.171253 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.171419 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":1092,"threads_started":1}
I20260929 02:11:02.171758 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.161295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:43063 (local address 127.0.0.1:42908)
0929 02:11:02.163234 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.163256 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:02.163383 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:02.164327 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:02.164334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:02.164361 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:11:02.164833 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.164849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.166639 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.166645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.168943 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.168957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.170823 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.170829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.171037 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.171085 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 02:11:02.171218 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 02:11:02.171402 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"thread_start_us":1110,"threads_started":1}
I20260929 02:11:02.172895 32313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.162532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38793 (local address 127.0.0.1:34602)
0929 02:11:02.164997 (+ 2465us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.165013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:02.165112 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:02.165752 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:02.165759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:02.165787 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:11:02.166296 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.166309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.168169 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.168174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.170079 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.170091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.172138 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.172144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.172332 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.172364 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:11:02.172455 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:11:02.172580 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2336,"mutex_wait_us":65,"thread_start_us":2120,"threads_started":1}
I20260929 02:11:02.173002 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.162967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.11.182.193:38793)
0929 02:11:02.163582 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.163590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.165240 (+ 1650us) server_negotiation.cc:407] Connection header received
0929 02:11:02.165346 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.165355 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.165557 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.165738 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:11:02.166468 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.168023 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.170266 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.171631 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.171787 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.172009 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.172530 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.172694 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":340,"threads_started":1}
I20260929 02:11:02.173794 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.160812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45847 (local address 127.0.0.1:34456)
0929 02:11:02.161807 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.161859 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:02.162074 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:02.165856 (+ 3782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:02.165860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:02.165885 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:11:02.166395 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.166407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.168220 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.168229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.170432 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.170447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.173240 (+ 2793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.173249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.173394 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.173416 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:11:02.173466 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 02:11:02.173544 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":109,"thread_start_us":891,"threads_started":1}
I20260929 02:11:02.174611 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.161046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.11.182.195:45847)
0929 02:11:02.165365 (+ 4319us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.165370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.165395 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:11:02.165462 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.165467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.165640 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.166055 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:11:02.166594 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.168106 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.170624 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.173030 (+ 2406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.174054 (+ 1024us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.174195 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.174246 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.174356 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4178,"thread_start_us":4025,"threads_started":1}
I20260929 02:11:02.175921 32102 tablet_service.cc:1467] Processing CreateTablet for tablet 491978ee84a4487fa91b7be77dafabbf (DEFAULT_TABLE table=test-table [id=addd85a666a64f4e88604afa8c470585]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:11:02.177517 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491978ee84a4487fa91b7be77dafabbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:02.178521 32232 tablet_service.cc:1467] Processing CreateTablet for tablet 491978ee84a4487fa91b7be77dafabbf (DEFAULT_TABLE table=test-table [id=addd85a666a64f4e88604afa8c470585]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:11:02.178521 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 491978ee84a4487fa91b7be77dafabbf (DEFAULT_TABLE table=test-table [id=addd85a666a64f4e88604afa8c470585]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:11:02.179983 32232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491978ee84a4487fa91b7be77dafabbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:02.179983 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491978ee84a4487fa91b7be77dafabbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:02.190711 32315 tablet_bootstrap.cc:492] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: Bootstrap starting.
I20260929 02:11:02.193413 32315 tablet_bootstrap.cc:654] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:02.194777 32315 log.cc:824] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:02.195981 32317 tablet_bootstrap.cc:492] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: Bootstrap starting.
I20260929 02:11:02.199067 32317 tablet_bootstrap.cc:654] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:02.200117 32317 log.cc:824] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:02.201431 32316 tablet_bootstrap.cc:492] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: Bootstrap starting.
I20260929 02:11:02.204032 32317 tablet_bootstrap.cc:492] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: No bootstrap required, opened a new log
I20260929 02:11:02.204330 32317 ts_tablet_manager.cc:1397] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:11:02.206192 32316 tablet_bootstrap.cc:654] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:02.208003 32316 log.cc:824] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:02.213678 32317 raft_consensus.cc:348] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } }
I20260929 02:11:02.214033 32317 raft_consensus.cc:374] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:02.214181 32317 raft_consensus.cc:729] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de122bec0d0e4f7a82081f24c8efed24, State: Initialized, Role: FOLLOWER
I20260929 02:11:02.215098 32317 consensus_queue.cc:260] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } }
I20260929 02:11:02.216840 32297 heartbeater.cc:503] Master 127.11.182.254:34091 was elected leader, sending a full tablet report...
I20260929 02:11:02.217191 32315 tablet_bootstrap.cc:492] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: No bootstrap required, opened a new log
I20260929 02:11:02.217502 32317 ts_tablet_manager.cc:1428] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 02:11:02.217507 32315 ts_tablet_manager.cc:1397] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.017s
I20260929 02:11:02.219285 32316 tablet_bootstrap.cc:492] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: No bootstrap required, opened a new log
I20260929 02:11:02.219610 32316 ts_tablet_manager.cc:1397] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260929 02:11:02.220890 32036 heartbeater.cc:503] Master 127.11.182.254:34091 was elected leader, sending a full tablet report...
I20260929 02:11:02.228483 32315 raft_consensus.cc:348] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } }
I20260929 02:11:02.228958 32315 raft_consensus.cc:374] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:02.229068 32315 raft_consensus.cc:729] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36b1baba1b24032bf9a714c05a3550b, State: Initialized, Role: FOLLOWER
I20260929 02:11:02.229748 32315 consensus_queue.cc:260] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } }
I20260929 02:11:02.231173 32167 heartbeater.cc:503] Master 127.11.182.254:34091 was elected leader, sending a full tablet report...
I20260929 02:11:02.231804 32315 ts_tablet_manager.cc:1428] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260929 02:11:02.231900 32316 raft_consensus.cc:348] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } }
I20260929 02:11:02.232358 32316 raft_consensus.cc:374] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:02.232491 32316 raft_consensus.cc:729] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb81501e3e9c400589972733d773294b, State: Initialized, Role: FOLLOWER
I20260929 02:11:02.233071 32316 consensus_queue.cc:260] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } }
I20260929 02:11:02.234653 32316 ts_tablet_manager.cc:1428] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260929 02:11:02.337790 32323 raft_consensus.cc:482] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:11:02.338124 32323 raft_consensus.cc:504] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } }
I20260929 02:11:02.339959 32323 leader_election.cc:288] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de122bec0d0e4f7a82081f24c8efed24 (127.11.182.195:45847), f36b1baba1b24032bf9a714c05a3550b (127.11.182.194:43063)
I20260929 02:11:02.346860 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.340339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:60233 (local address 127.11.182.194:43063)
0929 02:11:02.340546 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.340552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.341287 (+ 735us) server_negotiation.cc:407] Connection header received
0929 02:11:02.341513 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.341518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.341612 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.341717 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:11:02.342332 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.343136 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.345074 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.345904 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.346018 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.346091 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.346439 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.346568 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:11:02.346911 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.340563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.182.193:40401 (local address 127.11.182.195:45847)
0929 02:11:02.340768 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.340775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.341441 (+ 666us) server_negotiation.cc:407] Connection header received
0929 02:11:02.341550 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.341554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.341620 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.341716 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:11:02.342469 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.343276 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.345198 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.346008 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.346159 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.346228 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.346519 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.346654 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:11:02.346946 32325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.340223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.194:43063 (local address 127.11.182.193:60233)
0929 02:11:02.341130 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.341156 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:02.341341 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:02.341754 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:02.341759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:02.341788 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:11:02.342177 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.342187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.343256 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.343261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.344853 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.344865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.346090 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.346097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.346281 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.346307 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 02:11:02.346378 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 02:11:02.346500 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":4,"thread_start_us":737,"threads_started":1}
I20260929 02:11:02.346998 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.340345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.195:45847 (local address 127.11.182.193:40401)
0929 02:11:02.341300 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.341329 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:02.341444 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:02.341774 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:02.341777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:02.341793 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:11:02.342308 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.342322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.343411 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.343414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.344972 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:02.344988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.346207 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.346211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:02.346335 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.346378 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 02:11:02.346470 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:11:02.346584 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":53,"thread_start_us":382,"threads_started":1}
I20260929 02:11:02.348294 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491978ee84a4487fa91b7be77dafabbf" candidate_uuid: "fb81501e3e9c400589972733d773294b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36b1baba1b24032bf9a714c05a3550b" is_pre_election: true
I20260929 02:11:02.348300 32252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491978ee84a4487fa91b7be77dafabbf" candidate_uuid: "fb81501e3e9c400589972733d773294b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de122bec0d0e4f7a82081f24c8efed24" is_pre_election: true
I20260929 02:11:02.348873 32122 raft_consensus.cc:2454] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb81501e3e9c400589972733d773294b in term 0.
I20260929 02:11:02.348877 32252 raft_consensus.cc:2454] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb81501e3e9c400589972733d773294b in term 0.
W20260929 02:11:02.348966 32298 tablet.cc:2369] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:11:02.349558 31926 leader_election.cc:302] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f36b1baba1b24032bf9a714c05a3550b, fb81501e3e9c400589972733d773294b; no voters:
I20260929 02:11:02.349972 32323 raft_consensus.cc:2781] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:11:02.350121 32323 raft_consensus.cc:482] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:11:02.350246 32323 raft_consensus.cc:3037] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:02.352515 32323 raft_consensus.cc:504] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } }
I20260929 02:11:02.353071 32323 leader_election.cc:288] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [CANDIDATE]: Term 1 election: Requested vote from peers de122bec0d0e4f7a82081f24c8efed24 (127.11.182.195:45847), f36b1baba1b24032bf9a714c05a3550b (127.11.182.194:43063)
I20260929 02:11:02.353437 32252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491978ee84a4487fa91b7be77dafabbf" candidate_uuid: "fb81501e3e9c400589972733d773294b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de122bec0d0e4f7a82081f24c8efed24"
I20260929 02:11:02.353509 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491978ee84a4487fa91b7be77dafabbf" candidate_uuid: "fb81501e3e9c400589972733d773294b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36b1baba1b24032bf9a714c05a3550b"
I20260929 02:11:02.353633 32252 raft_consensus.cc:3037] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:02.353670 32122 raft_consensus.cc:3037] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:02.356807 32122 raft_consensus.cc:2454] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb81501e3e9c400589972733d773294b in term 1.
I20260929 02:11:02.356796 32252 raft_consensus.cc:2454] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb81501e3e9c400589972733d773294b in term 1.
I20260929 02:11:02.357189 31926 leader_election.cc:302] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f36b1baba1b24032bf9a714c05a3550b, fb81501e3e9c400589972733d773294b; no voters:
I20260929 02:11:02.357473 32323 raft_consensus.cc:2781] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:02.357908 32323 raft_consensus.cc:686] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 1 LEADER]: Becoming Leader. State: Replica: fb81501e3e9c400589972733d773294b, State: Running, Role: LEADER
I20260929 02:11:02.358377 32323 consensus_queue.cc:237] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } }
I20260929 02:11:02.362496 31850 catalog_manager.cc:5495] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b reported cstate change: term changed from 0 to 1, leader changed from <none> to fb81501e3e9c400589972733d773294b (127.11.182.193). New cstate: current_term: 1 leader_uuid: "fb81501e3e9c400589972733d773294b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb81501e3e9c400589972733d773294b" member_type: VOTER last_known_addr { host: "127.11.182.193" port: 38793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 } health_report { overall_health: UNKNOWN } } }
W20260929 02:11:02.391587 32168 tablet.cc:2369] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:11:02.408804 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.403750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.11.182.193:38793)
0929 02:11:02.403934 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.403938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.403958 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:11:02.404143 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.404148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.404241 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.404373 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:02.404680 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.405413 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.406736 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.407260 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.407366 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.407793 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:02.408315 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:02.408434 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.408494 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.408582 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:11:02.417295 32122 raft_consensus.cc:1261] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [term 1 FOLLOWER]: Refusing update from remote peer fb81501e3e9c400589972733d773294b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:11:02.417297 32252 raft_consensus.cc:1261] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [term 1 FOLLOWER]: Refusing update from remote peer fb81501e3e9c400589972733d773294b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:11:02.418336 32326 consensus_queue.cc:1035] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [LEADER]: Connected to new peer: Peer: permanent_uuid: "de122bec0d0e4f7a82081f24c8efed24" member_type: VOTER last_known_addr { host: "127.11.182.195" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:11:02.418702 32323 consensus_queue.cc:1035] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f36b1baba1b24032bf9a714c05a3550b" member_type: VOTER last_known_addr { host: "127.11.182.194" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:11:02.447098 31848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:11:02.447384 31848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:11:02.450831 31848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61ede32ff5fb465eaecb24522c7285b7: Sending DeleteTablet for 3 replicas of tablet 491978ee84a4487fa91b7be77dafabbf
I20260929 02:11:02.452181 32102 tablet_service.cc:1514] Processing DeleteTablet for tablet 491978ee84a4487fa91b7be77dafabbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:11:02 UTC) from {username='slave'} at 127.0.0.1:42908
I20260929 02:11:02.452647 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 31909
I20260929 02:11:02.452942 32232 tablet_service.cc:1514] Processing DeleteTablet for tablet 491978ee84a4487fa91b7be77dafabbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:11:02 UTC) from {username='slave'} at 127.0.0.1:34456
I20260929 02:11:02.453085 32340 tablet_replica.cc:330] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: stopping tablet replica
I20260929 02:11:02.453617 32340 raft_consensus.cc:2229] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:11:02.454360 31971 tablet_service.cc:1514] Processing DeleteTablet for tablet 491978ee84a4487fa91b7be77dafabbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:11:02 UTC) from {username='slave'} at 127.0.0.1:34602
I20260929 02:11:02.454428 32340 raft_consensus.cc:2258] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:11:02.455760 32341 tablet_replica.cc:330] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: stopping tablet replica
I20260929 02:11:02.456131 32341 raft_consensus.cc:2229] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:11:02.456581 32341 raft_consensus.cc:2258] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:11:02.457981 32340 ts_tablet_manager.cc:1905] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:11:02.462724 32340 ts_tablet_manager.cc:1918] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:11:02.462935 32340 log.cc:1196] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-1/wal/wals/491978ee84a4487fa91b7be77dafabbf
I20260929 02:11:02.464085 32340 ts_tablet_manager.cc:1939] T 491978ee84a4487fa91b7be77dafabbf P f36b1baba1b24032bf9a714c05a3550b: Deleting consensus metadata
I20260929 02:11:02.464736 32341 ts_tablet_manager.cc:1905] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:11:02.466063 31836 catalog_manager.cc:4841] TS f36b1baba1b24032bf9a714c05a3550b (127.11.182.194:43063): tablet 491978ee84a4487fa91b7be77dafabbf (table test-table [id=addd85a666a64f4e88604afa8c470585]) successfully deleted
I20260929 02:11:02.467406 32343 tablet_replica.cc:330] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: stopping tablet replica
I20260929 02:11:02.467820 32343 raft_consensus.cc:2229] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:11:02.468441 32343 raft_consensus.cc:2258] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:11:02.471104 32343 ts_tablet_manager.cc:1905] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:11:02.475066 32341 ts_tablet_manager.cc:1918] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:11:02.475267 32341 log.cc:1196] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-2/wal/wals/491978ee84a4487fa91b7be77dafabbf
I20260929 02:11:02.475828 32341 ts_tablet_manager.cc:1939] T 491978ee84a4487fa91b7be77dafabbf P de122bec0d0e4f7a82081f24c8efed24: Deleting consensus metadata
I20260929 02:11:02.476372 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.453683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.11.182.193:38793)
0929 02:11:02.454093 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.454101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.455166 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 02:11:02.464998 (+ 9832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.465006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.465121 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.465207 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:02.465923 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.467017 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.468486 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.469173 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.469313 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.474287 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:02.474307 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:02.474323 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:02.475216 (+ 893us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:02.475294 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:02.475328 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:02.475456 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.475742 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.475859 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260929 02:11:02.477710 31835 catalog_manager.cc:4841] TS de122bec0d0e4f7a82081f24c8efed24 (127.11.182.195:45847): tablet 491978ee84a4487fa91b7be77dafabbf (table test-table [id=addd85a666a64f4e88604afa8c470585]) successfully deleted
I20260929 02:11:02.477886 32031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:02.573832 32343 ts_tablet_manager.cc:1918] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:11:02.574190 32343 log.cc:1196] T 491978ee84a4487fa91b7be77dafabbf P fb81501e3e9c400589972733d773294b: Deleting WAL directory at /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647540175441-11995-0/minicluster-data/ts-0/wal/wals/491978ee84a4487fa91b7be77dafabbf
I20260929 02:11:02.574534 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 31909
I20260929 02:11:02.593802 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 32040
W20260929 02:11:02.593776 31835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.182.193:38793 (error 108)
I20260929 02:11:02.595752 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.595293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38793 (local address 127.0.0.1:34634)
0929 02:11:02.595460 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.595570 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 02:11:02.596033 31835 catalog_manager.cc:4584] TS fb81501e3e9c400589972733d773294b (127.11.182.193:38793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 491978ee84a4487fa91b7be77dafabbf: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38793: connect: Connection refused (error 111)
I20260929 02:11:02.600823 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.594604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.11.182.194:43063)
0929 02:11:02.594804 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.594811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.594900 (+ 89us) server_negotiation.cc:407] Connection header received
0929 02:11:02.595117 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.595121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.595204 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.595318 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:02.595694 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.596611 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.597872 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.598262 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.598417 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.599496 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:02.599517 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:02.599529 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:02.600092 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:02.600136 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:02.600158 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:02.600290 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.600420 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.600517 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:11:02.601537 32162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:02.649673 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.649205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38793 (local address 127.0.0.1:34644)
0929 02:11:02.649403 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.649493 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 02:11:02.650720 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.650364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38793 (local address 127.0.0.1:34650)
0929 02:11:02.650510 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.650585 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 02:11:02.689519 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 32040
I20260929 02:11:02.707119 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 32170
I20260929 02:11:02.711175 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.710625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38793 (local address 127.0.0.1:34652)
0929 02:11:02.710820 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.710986 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 02:11:02.712314 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.711856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38793 (local address 127.0.0.1:34668)
0929 02:11:02.712044 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.712145 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 02:11:02.714232 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.707952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.11.182.195:45847)
0929 02:11:02.708157 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:02.708162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:02.708182 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:11:02.708314 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:02.708318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:02.708411 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:02.708497 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:02.708868 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.709662 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.711177 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:02.711809 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:02.711947 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:02.712942 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:02.712951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:02.712966 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:02.713533 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:02.713580 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:02.713600 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:02.713692 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:02.713778 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 02:11:02.713859 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:11:02.715008 32292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:02.793007 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.792532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38793 (local address 127.0.0.1:34674)
0929 02:11:02.792730 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.792858 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:11:02.794216 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:02.793740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.193:38793 (local address 127.0.0.1:34690)
0929 02:11:02.793969 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:02.794068 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.193:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 02:11:02.804325 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 32170
I20260929 02:11:02.821415 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 31819
I20260929 02:11:02.822618 31880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:02.955338 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 31819
2026-09-29T02:11:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2799 ms)
[----------] 6 tests from EncryptionPolicyTest (21562 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:03 dist-test-slave-q06p krb5kdc[32351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:03 dist-test-slave-q06p krb5kdc[32351](info): set up 2 sockets
Sep 29 02:11:03 dist-test-slave-q06p krb5kdc[32351](info): commencing operation
krb5kdc: starting...
W20260929 02:11:04.075865 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:04 dist-test-slave-q06p krb5kdc[32351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:04.327080 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:04.649422 32364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:04.649694 32364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:04.649888 32364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:04.656955 32364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:04.657053 32364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:04.657099 32364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:04.657249 32364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:04.657333 32364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:04.666980 32364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:04.668296 32364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:04.669611 32364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:04.681295 32372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:04.681264 32370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:04.681346 32369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:04.682099 32364 server_base.cc:1034] running on GCE node
I20260929 02:11:04.682806 32364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:04.684185 32364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:04.685406 32364 hybrid_clock.cc:648] HybridClock initialized: now 1790647864685364 us; error 59 us; skew 500 ppm
I20260929 02:11:04.685747 32364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:04 dist-test-slave-q06p krb5kdc[32351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:04.688830 32364 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:04.690248 32364 webserver.cc:460] Webserver started at http://127.0.0.1:45447/ using document root <none> and password file <none>
I20260929 02:11:04.690840 32364 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:04.691020 32364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:04.691263 32364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:04.694021 32364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "cb68d397a90a4b9089f0bf9c6eb022f6"
format_stamp: "Formatted at 2026-09-29 02:11:04 on dist-test-slave-q06p"
server_key: "b77fc77cbb6c45c82581a23cb1274ba1"
server_key_iv: "c8bdcc99e384169178d4343d55eb6111"
server_key_version: "encryptionkey@0"
I20260929 02:11:04.694691 32364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "cb68d397a90a4b9089f0bf9c6eb022f6"
format_stamp: "Formatted at 2026-09-29 02:11:04 on dist-test-slave-q06p"
server_key: "b77fc77cbb6c45c82581a23cb1274ba1"
server_key_iv: "c8bdcc99e384169178d4343d55eb6111"
server_key_version: "encryptionkey@0"
I20260929 02:11:04.700345 32364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:11:04.703946 32379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:04.705579 32364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:11:04.705761 32364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "cb68d397a90a4b9089f0bf9c6eb022f6"
format_stamp: "Formatted at 2026-09-29 02:11:04 on dist-test-slave-q06p"
server_key: "b77fc77cbb6c45c82581a23cb1274ba1"
server_key_iv: "c8bdcc99e384169178d4343d55eb6111"
server_key_version: "encryptionkey@0"
I20260929 02:11:04.705947 32364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:04.725807 32364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:04.730031 32364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:04.730327 32364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:04.748883 32364 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38765
I20260929 02:11:04.748929 32430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38765 every 8 connection(s)
I20260929 02:11:04.750289 32364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:04.754753 32431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:04.759374 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 32364
I20260929 02:11:04.759590 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:04.760202 11995 external_mini_cluster.cc:1428] Setting key 9d55ed5691466fe20fab88169b0d618b
I20260929 02:11:04.766664 32431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6: Bootstrap starting.
I20260929 02:11:04.770458 32431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:04.771797 32431 log.cc:824] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:04.775341 32431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6: No bootstrap required, opened a new log
Sep 29 02:11:04 dist-test-slave-q06p krb5kdc[32351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:11:04.784019 32431 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb68d397a90a4b9089f0bf9c6eb022f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260929 02:11:04.784303 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:04.762076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55631 (local address 127.0.0.1:38765)
0929 02:11:04.762965 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:04.762979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:04.763019 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:11:04.764855 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:04.764876 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:04.765242 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:04.765543 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:04.767547 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:04.768794 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:04.770796 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:04.771534 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:04.771671 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:04.776749 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:04.776757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:04.776773 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:04.776804 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:04.780129 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:04.780900 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:04.780906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:04.780912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:04.781075 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:04.781535 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:04.781545 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:04.781547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:04.782104 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:04.782479 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:04.782981 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 02:11:04.783315 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":383,"threads_started":1}
I20260929 02:11:04.784432 32431 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:04.784544 32431 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb68d397a90a4b9089f0bf9c6eb022f6, State: Initialized, Role: FOLLOWER
I20260929 02:11:04.785148 32431 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb68d397a90a4b9089f0bf9c6eb022f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260929 02:11:04.785356 32431 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:04.785454 32431 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:04.785601 32431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:04.788256 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:04.788574 32431 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb68d397a90a4b9089f0bf9c6eb022f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260929 02:11:04.789073 32431 leader_election.cc:302] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb68d397a90a4b9089f0bf9c6eb022f6; no voters:
I20260929 02:11:04.789687 32431 leader_election.cc:288] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:04.789826 32436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:04.791229 32436 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [term 1 LEADER]: Becoming Leader. State: Replica: cb68d397a90a4b9089f0bf9c6eb022f6, State: Running, Role: LEADER
I20260929 02:11:04.791977 32436 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb68d397a90a4b9089f0bf9c6eb022f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260929 02:11:04.792171 32431 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:04.794966 32438 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb68d397a90a4b9089f0bf9c6eb022f6. Latest consensus state: current_term: 1 leader_uuid: "cb68d397a90a4b9089f0bf9c6eb022f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb68d397a90a4b9089f0bf9c6eb022f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } } }
I20260929 02:11:04.795323 32438 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:04.795435 32437 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb68d397a90a4b9089f0bf9c6eb022f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb68d397a90a4b9089f0bf9c6eb022f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } } }
I20260929 02:11:04.795852 32437 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:04.797745 32446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:04.803563 32446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:04.816882 32446 catalog_manager.cc:1347] Generated new cluster ID: 688de4e7ff814c2a8b230025207cf8b9
I20260929 02:11:04.816980 32446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:04.818063 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:04.794808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34011 (local address 127.0.0.1:38765)
0929 02:11:04.795181 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:04.795191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:04.798771 (+ 3580us) server_negotiation.cc:407] Connection header received
0929 02:11:04.799061 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:04.799067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:04.799147 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:04.799219 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:04.801305 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:04.802505 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:04.804268 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:04.804944 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:04.805040 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:04.808247 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:04.808254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:04.808259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:04.808297 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:04.812414 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:04.814596 (+ 2182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:04.814602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:04.814605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:04.814777 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:04.815497 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:04.815506 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:04.815512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:04.816091 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:04.816434 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:04.817284 (+ 850us) server_negotiation.cc:299] Negotiation successful
0929 02:11:04.817525 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 02:11:04.837670 32446 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:04.839684 32446 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:04.852275 32446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb68d397a90a4b9089f0bf9c6eb022f6: Generated new TSK 0
I20260929 02:11:04.853075 32446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:04.866935 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 32364
I20260929 02:11:04.868132 32426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:04.958137 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 32364
W20260929 02:11:04.975257 32349 connection.cc:535] client connection to 127.0.0.1:38765 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38765): Connection reset by peer (error 104)
2026-09-29T02:11:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:05 dist-test-slave-q06p krb5kdc[32464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:05 dist-test-slave-q06p krb5kdc[32464](info): set up 2 sockets
Sep 29 02:11:05 dist-test-slave-q06p krb5kdc[32464](info): commencing operation
krb5kdc: starting...
W20260929 02:11:06.055508 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:06 dist-test-slave-q06p krb5kdc[32464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:06.304183 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:06.631125 32477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:06.631414 32477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:06.631597 32477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:06.638810 32477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:06.638922 32477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:06.639024 32477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:06.639128 32477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:06.639199 32477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:06.648545 32477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:06.649709 32477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:06.651013 32477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:06.662601 32485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:06.662649 32482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:06.662673 32483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:06.663278 32477 server_base.cc:1034] running on GCE node
I20260929 02:11:06.664003 32477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:06.665395 32477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:06.666636 32477 hybrid_clock.cc:648] HybridClock initialized: now 1790647866666591 us; error 56 us; skew 500 ppm
I20260929 02:11:06.667040 32477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:06 dist-test-slave-q06p krb5kdc[32464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:06.669986 32477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:06.671418 32477 webserver.cc:460] Webserver started at http://127.0.0.1:39987/ using document root <none> and password file <none>
I20260929 02:11:06.672065 32477 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:06.672196 32477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:06.672473 32477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:06.675428 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "df1f21a6664443ce897a6e419236839e"
format_stamp: "Formatted at 2026-09-29 02:11:06 on dist-test-slave-q06p"
server_key: "82f709ca8f684bf4cc5ba34483e3b7e7"
server_key_iv: "aa587c85b5a141c36871be7038c45bde"
server_key_version: "encryptionkey@0"
I20260929 02:11:06.676182 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "df1f21a6664443ce897a6e419236839e"
format_stamp: "Formatted at 2026-09-29 02:11:06 on dist-test-slave-q06p"
server_key: "82f709ca8f684bf4cc5ba34483e3b7e7"
server_key_iv: "aa587c85b5a141c36871be7038c45bde"
server_key_version: "encryptionkey@0"
I20260929 02:11:06.681008 32477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:11:06.684360 32492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:06.685577 32477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:11:06.685736 32477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "df1f21a6664443ce897a6e419236839e"
format_stamp: "Formatted at 2026-09-29 02:11:06 on dist-test-slave-q06p"
server_key: "82f709ca8f684bf4cc5ba34483e3b7e7"
server_key_iv: "aa587c85b5a141c36871be7038c45bde"
server_key_version: "encryptionkey@0"
I20260929 02:11:06.685917 32477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:06.706369 32477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:06.710670 32477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:06.711010 32477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:06.728919 32477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34809
I20260929 02:11:06.728955 32543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34809 every 8 connection(s)
I20260929 02:11:06.730203 32477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:06.733618 32544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:06.737485 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 32477
I20260929 02:11:06.737668 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:06.738209 11995 external_mini_cluster.cc:1428] Setting key a8dd23e0a54261dee671896ea9c99dcd
I20260929 02:11:06.746294 32544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e: Bootstrap starting.
I20260929 02:11:06.749889 32544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:06.751097 32544 log.cc:824] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:06.751377 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:06.739984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43999 (local address 127.0.0.1:34809)
0929 02:11:06.740808 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:06.740819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:06.740858 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:11:06.742394 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:06.742415 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:06.742792 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:06.743165 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:06.743718 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:06.745070 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:06.746824 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:06.747497 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:06.747623 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:06.748942 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:06.748949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:06.748962 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:06.749481 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:06.749548 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:06.749575 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:06.749705 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:06.750159 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 02:11:06.750303 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":418,"threads_started":1}
I20260929 02:11:06.753973 32544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e: No bootstrap required, opened a new log
I20260929 02:11:06.755214 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:06.762617 32544 raft_consensus.cc:348] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1f21a6664443ce897a6e419236839e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } }
I20260929 02:11:06.763090 32544 raft_consensus.cc:374] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:06.763191 32544 raft_consensus.cc:729] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1f21a6664443ce897a6e419236839e, State: Initialized, Role: FOLLOWER
I20260929 02:11:06.763800 32544 consensus_queue.cc:260] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1f21a6664443ce897a6e419236839e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } }
I20260929 02:11:06.764073 32544 raft_consensus.cc:388] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:06.764249 32544 raft_consensus.cc:482] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:06.764415 32544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:06.766042 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:06.759289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41607 (local address 127.0.0.1:34809)
0929 02:11:06.759483 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:06.759491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:06.759995 (+ 504us) server_negotiation.cc:407] Connection header received
0929 02:11:06.760148 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:06.760152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:06.760211 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:06.760299 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:06.760842 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:06.761614 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:06.763494 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:06.763971 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:06.764119 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:06.765213 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:06.765218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:06.765221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:06.765237 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:06.765304 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:06.765320 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:06.765416 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:06.765633 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 02:11:06.765713 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:11:06.767346 32544 raft_consensus.cc:504] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1f21a6664443ce897a6e419236839e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } }
I20260929 02:11:06.767822 32544 leader_election.cc:302] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df1f21a6664443ce897a6e419236839e; no voters:
I20260929 02:11:06.768342 32544 leader_election.cc:288] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:06.768482 32554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:06.769153 32554 raft_consensus.cc:686] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [term 1 LEADER]: Becoming Leader. State: Replica: df1f21a6664443ce897a6e419236839e, State: Running, Role: LEADER
I20260929 02:11:06.769657 32554 consensus_queue.cc:237] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1f21a6664443ce897a6e419236839e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } }
I20260929 02:11:06.770335 32544 sys_catalog.cc:564] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:06.772348 32555 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df1f21a6664443ce897a6e419236839e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1f21a6664443ce897a6e419236839e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } } }
I20260929 02:11:06.772826 32555 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:06.772385 32556 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1f21a6664443ce897a6e419236839e. Latest consensus state: current_term: 1 leader_uuid: "df1f21a6664443ce897a6e419236839e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1f21a6664443ce897a6e419236839e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } } }
I20260929 02:11:06.773038 32556 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:06.773706 32559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:06.779625 32559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:06.790372 32559 catalog_manager.cc:1347] Generated new cluster ID: 7ded4f82e79b4891b5f5fce706bab941
I20260929 02:11:06.790526 32559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:06.804922 32559 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:06.806800 32559 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:06.822186 32559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df1f21a6664443ce897a6e419236839e: Generated new TSK 0
I20260929 02:11:06.823252 32559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:06.840104 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 32477
W20260929 02:11:06.841630 32539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43999
W20260929 02:11:06.842334 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 32477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:06.842509 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 32477
W20260929 02:11:06.859056 32462 connection.cc:535] client connection to 127.0.0.1:34809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34809): Connection reset by peer (error 104)
2026-09-29T02:11:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:06 dist-test-slave-q06p krb5kdc[32575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:06 dist-test-slave-q06p krb5kdc[32575](info): set up 2 sockets
Sep 29 02:11:06 dist-test-slave-q06p krb5kdc[32575](info): commencing operation
krb5kdc: starting...
W20260929 02:11:07.951638 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:08 dist-test-slave-q06p krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:08.198719 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:08.525174 32588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:08.525453 32588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:08.525609 32588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:08.533077 32588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:08.533180 32588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:08.533277 32588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:08.533394 32588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:08.533461 32588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:08.543203 32588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:08.544523 32588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:08.545782 32588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:08.557278 32596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:08.557360 32593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:08.557412 32594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:08.558215 32588 server_base.cc:1034] running on GCE node
I20260929 02:11:08.559087 32588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:08.560468 32588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:08.561695 32588 hybrid_clock.cc:648] HybridClock initialized: now 1790647868561653 us; error 38 us; skew 500 ppm
I20260929 02:11:08.562039 32588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:08 dist-test-slave-q06p krb5kdc[32575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:08.569568 32588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:08.571156 32588 webserver.cc:460] Webserver started at http://127.0.0.1:35387/ using document root <none> and password file <none>
I20260929 02:11:08.571885 32588 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:08.572026 32588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:08.572319 32588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:08.575309 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "57491d34510e4dda88f1885bbabfe0ff"
format_stamp: "Formatted at 2026-09-29 02:11:08 on dist-test-slave-q06p"
server_key: "2f46d3b92ef88750fbb7d6aed06750d8"
server_key_iv: "2ebb70e96d1b0915a044e48eb30b744b"
server_key_version: "encryptionkey@0"
I20260929 02:11:08.576094 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "57491d34510e4dda88f1885bbabfe0ff"
format_stamp: "Formatted at 2026-09-29 02:11:08 on dist-test-slave-q06p"
server_key: "2f46d3b92ef88750fbb7d6aed06750d8"
server_key_iv: "2ebb70e96d1b0915a044e48eb30b744b"
server_key_version: "encryptionkey@0"
I20260929 02:11:08.581228 32588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 02:11:08.584663 32603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:08.585994 32588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:11:08.586165 32588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "57491d34510e4dda88f1885bbabfe0ff"
format_stamp: "Formatted at 2026-09-29 02:11:08 on dist-test-slave-q06p"
server_key: "2f46d3b92ef88750fbb7d6aed06750d8"
server_key_iv: "2ebb70e96d1b0915a044e48eb30b744b"
server_key_version: "encryptionkey@0"
I20260929 02:11:08.586380 32588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:08.592232 32588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:08.596464 32588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:08.596787 32588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:08.615065 32588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39509
I20260929 02:11:08.615113 32654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39509 every 8 connection(s)
I20260929 02:11:08.616394 32588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:08.620069 32655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:08.622872 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 32588
I20260929 02:11:08.623054 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:08.623557 11995 external_mini_cluster.cc:1428] Setting key 056cf99304d2ad7ad19dfc84fa4d7af2
I20260929 02:11:08.631146 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:08.625287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53541 (local address 127.0.0.1:39509)
0929 02:11:08.626063 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:08.626075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:08.626114 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:11:08.627790 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:08.627812 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:08.628145 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:08.628488 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:08.629477 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:08.629486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:08.629498 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:08.629514 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:08.629571 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:08.629589 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:08.629784 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:08.629942 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 02:11:08.630147 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":369,"threads_started":1}
I20260929 02:11:08.632272 32655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff: Bootstrap starting.
I20260929 02:11:08.634704 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:08.635619 32655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:08.636613 32655 log.cc:824] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:08.639514 32655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff: No bootstrap required, opened a new log
I20260929 02:11:08.641481 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:08.639094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37859 (local address 127.0.0.1:39509)
0929 02:11:08.639300 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:08.639306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:08.639616 (+ 310us) server_negotiation.cc:407] Connection header received
0929 02:11:08.639768 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:08.639772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:08.639817 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:08.639893 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:08.640827 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:08.640831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:08.640834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:08.640851 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:08.640887 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:08.640902 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:08.640984 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:08.641127 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 02:11:08.641170 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:11:08.647918 32655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57491d34510e4dda88f1885bbabfe0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260929 02:11:08.648274 32655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:08.648386 32655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57491d34510e4dda88f1885bbabfe0ff, State: Initialized, Role: FOLLOWER
I20260929 02:11:08.649120 32655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57491d34510e4dda88f1885bbabfe0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260929 02:11:08.649406 32665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:11:08.649646 32665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57491d34510e4dda88f1885bbabfe0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260929 02:11:08.649366 32655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:08.650238 32655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:08.650228 32665 leader_election.cc:302] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57491d34510e4dda88f1885bbabfe0ff; no voters:
I20260929 02:11:08.650404 32655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:08.650964 32665 leader_election.cc:288] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260929 02:11:08.652863 32655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57491d34510e4dda88f1885bbabfe0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260929 02:11:08.653115 32655 leader_election.cc:302] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57491d34510e4dda88f1885bbabfe0ff; no voters:
I20260929 02:11:08.653158 32666 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 02:11:08.653421 32655 leader_election.cc:288] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:08.653573 32665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:08.654750 32665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [term 1 LEADER]: Becoming Leader. State: Replica: 57491d34510e4dda88f1885bbabfe0ff, State: Running, Role: LEADER
I20260929 02:11:08.655232 32665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57491d34510e4dda88f1885bbabfe0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260929 02:11:08.655821 32655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:08.663650 32667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57491d34510e4dda88f1885bbabfe0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57491d34510e4dda88f1885bbabfe0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } } }
I20260929 02:11:08.664078 32667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:08.663973 32668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57491d34510e4dda88f1885bbabfe0ff. Latest consensus state: current_term: 1 leader_uuid: "57491d34510e4dda88f1885bbabfe0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57491d34510e4dda88f1885bbabfe0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } } }
I20260929 02:11:08.664240 32668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff [sys.catalog]: This master's current role is: LEADER
W20260929 02:11:08.672273 32680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:11:08.672428 32680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:11:08.672573 32681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:08.674074 32681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:08.682444 32681 catalog_manager.cc:1347] Generated new cluster ID: 1cfcd4ceee8b4f93bf48b99289634f6e
I20260929 02:11:08.682564 32681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:08.716041 32681 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:08.716569 32681 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:08.732069 32681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57491d34510e4dda88f1885bbabfe0ff: Generated new TSK 0
I20260929 02:11:08.733120 32681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:08.738631 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 32588
W20260929 02:11:08.739727 32650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53541
W20260929 02:11:08.740445 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 32588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:08.740619 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 32588
2026-09-29T02:11:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:08 dist-test-slave-q06p krb5kdc[32687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:08 dist-test-slave-q06p krb5kdc[32687](info): set up 2 sockets
Sep 29 02:11:08 dist-test-slave-q06p krb5kdc[32687](info): commencing operation
krb5kdc: starting...
W20260929 02:11:09.859933 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:10 dist-test-slave-q06p krb5kdc[32687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:10.106220 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:10.429106 32700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:10.429396 32700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:10.429551 32700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:10.436792 32700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:10.436945 32700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:10.437043 32700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:10.437139 32700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:10.437212 32700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:10.447253 32700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:10.448593 32700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:10.449899 32700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:10.461422 32705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:10.461452 32706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:10.461501 32708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:10.462266 32700 server_base.cc:1034] running on GCE node
I20260929 02:11:10.463070 32700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:10.464426 32700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:10.465631 32700 hybrid_clock.cc:648] HybridClock initialized: now 1790647870465584 us; error 55 us; skew 500 ppm
I20260929 02:11:10.465956 32700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:10 dist-test-slave-q06p krb5kdc[32687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:10.468964 32700 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:10.470250 32700 webserver.cc:460] Webserver started at http://127.0.0.1:45621/ using document root <none> and password file <none>
I20260929 02:11:10.470831 32700 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:10.470995 32700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:10.471244 32700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:10.474015 32700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "2062db568a144b32b3c3bf5cb93eb2d7"
format_stamp: "Formatted at 2026-09-29 02:11:10 on dist-test-slave-q06p"
server_key: "b6c1702df937f1ffbd0c9562c138367f"
server_key_iv: "37e327bee63187af443bb18e93d6de7f"
server_key_version: "encryptionkey@0"
I20260929 02:11:10.474713 32700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "2062db568a144b32b3c3bf5cb93eb2d7"
format_stamp: "Formatted at 2026-09-29 02:11:10 on dist-test-slave-q06p"
server_key: "b6c1702df937f1ffbd0c9562c138367f"
server_key_iv: "37e327bee63187af443bb18e93d6de7f"
server_key_version: "encryptionkey@0"
I20260929 02:11:10.479992 32700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 02:11:10.483361 32715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:10.484647 32700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:11:10.484826 32700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "2062db568a144b32b3c3bf5cb93eb2d7"
format_stamp: "Formatted at 2026-09-29 02:11:10 on dist-test-slave-q06p"
server_key: "b6c1702df937f1ffbd0c9562c138367f"
server_key_iv: "37e327bee63187af443bb18e93d6de7f"
server_key_version: "encryptionkey@0"
I20260929 02:11:10.484994 32700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:10.503118 32700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:10.506742 32700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:10.507027 32700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:10.526397 32700 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38991
I20260929 02:11:10.526448 32766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38991 every 8 connection(s)
I20260929 02:11:10.528285 32700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:10.529161 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 32700
I20260929 02:11:10.529349 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:10.529919 11995 external_mini_cluster.cc:1428] Setting key 9ceb5a07d31ddbd59726bf48eb121c55
I20260929 02:11:10.532912 32767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:10.546994 32767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7: Bootstrap starting.
Sep 29 02:11:10 dist-test-slave-q06p krb5kdc[32687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:11:10.550829 32767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:10.552203 32767 log.cc:824] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:10.555831 32767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7: No bootstrap required, opened a new log
I20260929 02:11:10.557375 302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:10.532113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46783 (local address 127.0.0.1:38991)
0929 02:11:10.533054 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:10.533070 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:10.535449 (+ 2379us) server_negotiation.cc:407] Connection header received
0929 02:11:10.537576 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:10.537604 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:10.538122 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:10.538615 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:10.539896 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:10.541937 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:10.543230 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:10.544318 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:10.544501 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:10.548627 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:10.548638 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:10.548658 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:10.548713 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:10.552647 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:10.553533 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:10.553540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:10.553546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:10.553796 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:10.554192 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:10.554197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:10.554201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:10.554925 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:10.555242 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:10.555523 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 02:11:10.555995 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":419,"threads_started":1}
I20260929 02:11:10.561234 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:10.564795 32767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062db568a144b32b3c3bf5cb93eb2d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } }
I20260929 02:11:10.565145 32767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:10.565260 32767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2062db568a144b32b3c3bf5cb93eb2d7, State: Initialized, Role: FOLLOWER
I20260929 02:11:10.565968 32767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062db568a144b32b3c3bf5cb93eb2d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } }
I20260929 02:11:10.566251 32767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:10.566418 32767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:10.566613 32767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:10.568881 32767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062db568a144b32b3c3bf5cb93eb2d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } }
I20260929 02:11:10.569363 32767 leader_election.cc:302] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2062db568a144b32b3c3bf5cb93eb2d7; no voters:
I20260929 02:11:10.570035 32767 leader_election.cc:288] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:10.570149 309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:10.572693 32767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:10.576874 309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [term 1 LEADER]: Becoming Leader. State: Replica: 2062db568a144b32b3c3bf5cb93eb2d7, State: Running, Role: LEADER
I20260929 02:11:10.577555 309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062db568a144b32b3c3bf5cb93eb2d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } }
I20260929 02:11:10.579643 302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:10.565526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45441 (local address 127.0.0.1:38991)
0929 02:11:10.565717 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:10.565723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:10.566185 (+ 462us) server_negotiation.cc:407] Connection header received
0929 02:11:10.566378 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:10.566383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:10.566458 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:10.566551 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:10.567993 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:10.568700 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:10.569994 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:10.570801 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:10.570980 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:10.573589 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:10.573594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:10.573598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:10.573628 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:10.576888 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:10.577522 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:10.577529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:10.577536 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:10.577691 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:10.578069 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:10.578075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:10.578078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:10.578436 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:10.578724 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:10.579067 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 02:11:10.579282 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:11:10.589274 311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2062db568a144b32b3c3bf5cb93eb2d7. Latest consensus state: current_term: 1 leader_uuid: "2062db568a144b32b3c3bf5cb93eb2d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062db568a144b32b3c3bf5cb93eb2d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } } }
I20260929 02:11:10.589646 311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:10.590157 310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2062db568a144b32b3c3bf5cb93eb2d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062db568a144b32b3c3bf5cb93eb2d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } } }
I20260929 02:11:10.590525 310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:10.590837 325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:10.595054 325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:10.603242 325 catalog_manager.cc:1347] Generated new cluster ID: 1d9fe83c03884a38bde13b095e789d60
I20260929 02:11:10.603367 325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:10.614058 325 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:10.615918 325 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:10.626092 325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2062db568a144b32b3c3bf5cb93eb2d7: Generated new TSK 0
I20260929 02:11:10.626942 325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:10.640413 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 32700
I20260929 02:11:10.641618 32762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:10.732213 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 32700
2026-09-29T02:11:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:10 dist-test-slave-q06p krb5kdc[332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:10 dist-test-slave-q06p krb5kdc[332](info): set up 2 sockets
Sep 29 02:11:10 dist-test-slave-q06p krb5kdc[332](info): commencing operation
krb5kdc: starting...
W20260929 02:11:11.836076 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:11 dist-test-slave-q06p krb5kdc[332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:12.088028 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:12.409924 345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:12.410230 345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:12.410398 345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:12.417598 345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:12.417706 345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:12.417749 345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:12.417840 345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:12.417922 345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:12.427678 345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:12.428916 345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:12.430184 345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:12.442297 351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:12.442358 350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:12.442380 353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:12.443087 345 server_base.cc:1034] running on GCE node
I20260929 02:11:12.443871 345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:12.445259 345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:12.446655 345 hybrid_clock.cc:648] HybridClock initialized: now 1790647872446600 us; error 54 us; skew 500 ppm
I20260929 02:11:12.447154 345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:12 dist-test-slave-q06p krb5kdc[332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:12.450282 345 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:12.452108 345 webserver.cc:460] Webserver started at http://127.0.0.1:40863/ using document root <none> and password file <none>
I20260929 02:11:12.452760 345 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:12.452858 345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:12.453107 345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:12.456075 345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "a244faaf1e4a4b4895fd4e754b2a5df8"
format_stamp: "Formatted at 2026-09-29 02:11:12 on dist-test-slave-q06p"
server_key: "f9959adde8e7d06c3bde7b17bfeb3f91"
server_key_iv: "58f28c34d92b7a39d0bfc3ad90b42546"
server_key_version: "encryptionkey@0"
I20260929 02:11:12.456780 345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "a244faaf1e4a4b4895fd4e754b2a5df8"
format_stamp: "Formatted at 2026-09-29 02:11:12 on dist-test-slave-q06p"
server_key: "f9959adde8e7d06c3bde7b17bfeb3f91"
server_key_iv: "58f28c34d92b7a39d0bfc3ad90b42546"
server_key_version: "encryptionkey@0"
I20260929 02:11:12.461823 345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:11:12.465067 360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:12.466300 345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:11:12.466429 345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "a244faaf1e4a4b4895fd4e754b2a5df8"
format_stamp: "Formatted at 2026-09-29 02:11:12 on dist-test-slave-q06p"
server_key: "f9959adde8e7d06c3bde7b17bfeb3f91"
server_key_iv: "58f28c34d92b7a39d0bfc3ad90b42546"
server_key_version: "encryptionkey@0"
I20260929 02:11:12.466596 345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:12.489187 345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:12.493221 345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:12.493454 345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:12.511363 345 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40885
I20260929 02:11:12.511402 411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40885 every 8 connection(s)
I20260929 02:11:12.512660 345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:12.516186 412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:12.522472 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 345
I20260929 02:11:12.522660 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:12.523213 11995 external_mini_cluster.cc:1428] Setting key d3bfb0f7c2cdfa4611f4513d95c115bb
I20260929 02:11:12.526829 412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8: Bootstrap starting.
I20260929 02:11:12.530639 412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:12.531978 412 log.cc:824] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:12.535817 412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8: No bootstrap required, opened a new log
I20260929 02:11:12.538165 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:12.524937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40875 (local address 127.0.0.1:40885)
0929 02:11:12.525703 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:12.525714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:12.525752 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:11:12.527540 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:12.527563 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:12.527944 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:12.528268 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:12.528898 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:12.530245 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:12.532212 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:12.533282 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:12.533484 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:12.535321 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:12.535333 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:12.535353 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:12.536069 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:12.536173 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:12.536208 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:12.536328 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:12.536938 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 02:11:12.537142 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":344,"threads_started":1}
I20260929 02:11:12.541792 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:12.544111 412 raft_consensus.cc:348] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a244faaf1e4a4b4895fd4e754b2a5df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40885 } }
I20260929 02:11:12.544544 412 raft_consensus.cc:374] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:12.544659 412 raft_consensus.cc:729] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a244faaf1e4a4b4895fd4e754b2a5df8, State: Initialized, Role: FOLLOWER
I20260929 02:11:12.545265 412 consensus_queue.cc:260] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a244faaf1e4a4b4895fd4e754b2a5df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40885 } }
I20260929 02:11:12.545504 412 raft_consensus.cc:388] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:12.545660 412 raft_consensus.cc:482] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:12.545838 412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:12.549121 412 raft_consensus.cc:504] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a244faaf1e4a4b4895fd4e754b2a5df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40885 } }
I20260929 02:11:12.549608 412 leader_election.cc:302] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a244faaf1e4a4b4895fd4e754b2a5df8; no voters:
I20260929 02:11:12.550238 412 leader_election.cc:288] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:12.550372 422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:12.551829 422 raft_consensus.cc:686] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [term 1 LEADER]: Becoming Leader. State: Replica: a244faaf1e4a4b4895fd4e754b2a5df8, State: Running, Role: LEADER
I20260929 02:11:12.552389 422 consensus_queue.cc:237] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a244faaf1e4a4b4895fd4e754b2a5df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40885 } }
I20260929 02:11:12.552596 412 sys_catalog.cc:564] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:12.554363 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:12.546106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47381 (local address 127.0.0.1:40885)
0929 02:11:12.546743 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:12.546750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:12.547069 (+ 319us) server_negotiation.cc:407] Connection header received
0929 02:11:12.547206 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:12.547211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:12.547294 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:12.547400 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:12.547876 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:12.548925 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:12.550484 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:12.551240 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:12.552007 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:12.552609 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:12.552618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:12.552625 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:12.552645 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:12.552692 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:12.552719 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:12.553690 (+ 971us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:12.553744 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 02:11:12.553861 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514}
I20260929 02:11:12.554875 424 sys_catalog.cc:455] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a244faaf1e4a4b4895fd4e754b2a5df8. Latest consensus state: current_term: 1 leader_uuid: "a244faaf1e4a4b4895fd4e754b2a5df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a244faaf1e4a4b4895fd4e754b2a5df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40885 } } }
I20260929 02:11:12.555256 424 sys_catalog.cc:458] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:12.555791 423 sys_catalog.cc:455] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a244faaf1e4a4b4895fd4e754b2a5df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a244faaf1e4a4b4895fd4e754b2a5df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40885 } } }
I20260929 02:11:12.556044 423 sys_catalog.cc:458] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:12.557523 427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:12.562791 427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:12.572772 427 catalog_manager.cc:1347] Generated new cluster ID: 76bae6b04f02496d928d8a0fdfbb39f6
I20260929 02:11:12.572901 427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:12.585062 427 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:12.586333 427 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:12.595252 427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a244faaf1e4a4b4895fd4e754b2a5df8: Generated new TSK 0
I20260929 02:11:12.595990 427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:12.603108 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 345
W20260929 02:11:12.604489 407 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40875
W20260929 02:11:12.605074 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 345: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:12.605227 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 345
2026-09-29T02:11:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:12 dist-test-slave-q06p krb5kdc[443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:12 dist-test-slave-q06p krb5kdc[443](info): set up 2 sockets
Sep 29 02:11:12 dist-test-slave-q06p krb5kdc[443](info): commencing operation
krb5kdc: starting...
W20260929 02:11:13.715828 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:13 dist-test-slave-q06p krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:13.974314 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:14.296023 456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:14.296365 456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:14.296514 456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:14.303915 456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:14.304057 456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:14.304163 456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:14.304265 456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:14.304337 456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:14.313935 456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:14.315196 456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:14.316453 456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:14.327678 462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:14.327745 461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:14.327775 464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:14.328649 456 server_base.cc:1034] running on GCE node
I20260929 02:11:14.329433 456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:14.330883 456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:14.332108 456 hybrid_clock.cc:648] HybridClock initialized: now 1790647874332070 us; error 34 us; skew 500 ppm
I20260929 02:11:14.332439 456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:14 dist-test-slave-q06p krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:14.341149 456 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:14.342710 456 webserver.cc:460] Webserver started at http://127.0.0.1:45693/ using document root <none> and password file <none>
I20260929 02:11:14.343389 456 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:14.343523 456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:14.343812 456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:14.346588 456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "ba1bb3b77bf84811aea2739e5f89171a"
format_stamp: "Formatted at 2026-09-29 02:11:14 on dist-test-slave-q06p"
server_key: "192f59830efec600f79d9da3d0fab779"
server_key_iv: "a69a882b121ce0678de0b41295f4c490"
server_key_version: "encryptionkey@0"
I20260929 02:11:14.347398 456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "ba1bb3b77bf84811aea2739e5f89171a"
format_stamp: "Formatted at 2026-09-29 02:11:14 on dist-test-slave-q06p"
server_key: "192f59830efec600f79d9da3d0fab779"
server_key_iv: "a69a882b121ce0678de0b41295f4c490"
server_key_version: "encryptionkey@0"
I20260929 02:11:14.352165 456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 02:11:14.355396 471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:14.356596 456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:11:14.356755 456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "ba1bb3b77bf84811aea2739e5f89171a"
format_stamp: "Formatted at 2026-09-29 02:11:14 on dist-test-slave-q06p"
server_key: "192f59830efec600f79d9da3d0fab779"
server_key_iv: "a69a882b121ce0678de0b41295f4c490"
server_key_version: "encryptionkey@0"
I20260929 02:11:14.356930 456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:14.362524 456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:14.366079 456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:14.366349 456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:14.384387 456 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44637
I20260929 02:11:14.384438 522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44637 every 8 connection(s)
I20260929 02:11:14.385735 456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:14.387928 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 456
I20260929 02:11:14.388175 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:14.388682 11995 external_mini_cluster.cc:1428] Setting key 330573a924d4ec2addb7b789fad09d53
I20260929 02:11:14.389448 523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:14.396390 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:14.390461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34753 (local address 127.0.0.1:44637)
0929 02:11:14.391331 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:14.391344 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:14.391385 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:11:14.392963 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:14.392986 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:14.393342 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:14.393698 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:14.394713 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:14.394719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:14.394732 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:14.394763 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:14.394833 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:14.394918 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:14.395109 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:14.395221 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 02:11:14.395425 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":394,"threads_started":1}
I20260929 02:11:14.399916 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:14.402058 523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a: Bootstrap starting.
I20260929 02:11:14.405525 523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:14.406412 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:14.404109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46827 (local address 127.0.0.1:44637)
0929 02:11:14.404297 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:14.404302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:14.404566 (+ 264us) server_negotiation.cc:407] Connection header received
0929 02:11:14.404731 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:14.404735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:14.404776 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:14.404849 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:14.405782 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:14.405786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:14.405789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:14.405804 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:14.405839 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:14.405856 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:14.405946 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:14.406085 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 02:11:14.406127 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:11:14.406804 523 log.cc:824] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:14.409346 523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a: No bootstrap required, opened a new log
I20260929 02:11:14.416447 523 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1bb3b77bf84811aea2739e5f89171a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44637 } }
I20260929 02:11:14.416841 523 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:14.416944 523 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1bb3b77bf84811aea2739e5f89171a, State: Initialized, Role: FOLLOWER
I20260929 02:11:14.417495 523 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1bb3b77bf84811aea2739e5f89171a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44637 } }
I20260929 02:11:14.417742 523 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:14.417881 523 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:14.418041 523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:14.420326 523 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1bb3b77bf84811aea2739e5f89171a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44637 } }
I20260929 02:11:14.420817 523 leader_election.cc:302] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba1bb3b77bf84811aea2739e5f89171a; no voters:
I20260929 02:11:14.421384 523 leader_election.cc:288] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:14.421507 533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:14.422309 533 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [term 1 LEADER]: Becoming Leader. State: Replica: ba1bb3b77bf84811aea2739e5f89171a, State: Running, Role: LEADER
I20260929 02:11:14.422885 533 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1bb3b77bf84811aea2739e5f89171a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44637 } }
I20260929 02:11:14.423574 523 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:14.425398 534 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba1bb3b77bf84811aea2739e5f89171a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1bb3b77bf84811aea2739e5f89171a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44637 } } }
I20260929 02:11:14.425588 535 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba1bb3b77bf84811aea2739e5f89171a. Latest consensus state: current_term: 1 leader_uuid: "ba1bb3b77bf84811aea2739e5f89171a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1bb3b77bf84811aea2739e5f89171a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44637 } } }
I20260929 02:11:14.425863 535 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:14.425863 534 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:14.430099 539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:14.436633 539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:14.444783 539 catalog_manager.cc:1347] Generated new cluster ID: f83fd34f751448578d19141e4f1793fb
I20260929 02:11:14.444905 539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:14.487818 539 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:14.488245 539 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:14.502152 539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba1bb3b77bf84811aea2739e5f89171a: Generated new TSK 0
I20260929 02:11:14.503145 539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:14.525326 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 456
W20260929 02:11:14.526522 518 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34753
W20260929 02:11:14.527310 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 456: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:14.527451 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 456
2026-09-29T02:11:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:14 dist-test-slave-q06p krb5kdc[554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:14 dist-test-slave-q06p krb5kdc[554](info): set up 2 sockets
Sep 29 02:11:14 dist-test-slave-q06p krb5kdc[554](info): commencing operation
krb5kdc: starting...
W20260929 02:11:15.643957 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:15 dist-test-slave-q06p krb5kdc[554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:15.909852 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:16.251408 567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:16.251694 567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:16.251785 567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:16.259145 567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:16.259269 567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:16.259315 567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:16.259369 567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:16.259413 567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:16.269486 567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:16.270771 567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:16.272006 567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:16.283416 572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:16.283365 573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:16.283588 575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:16.284451 567 server_base.cc:1034] running on GCE node
I20260929 02:11:16.285208 567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:16.286504 567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:16.287699 567 hybrid_clock.cc:648] HybridClock initialized: now 1790647876287668 us; error 33 us; skew 500 ppm
I20260929 02:11:16.287988 567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:16 dist-test-slave-q06p krb5kdc[554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:16.291077 567 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:16.292553 567 webserver.cc:460] Webserver started at http://127.0.0.1:43361/ using document root <none> and password file <none>
I20260929 02:11:16.293202 567 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:16.293335 567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:16.293653 567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:16.296758 567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "20698ad91b244d8d9bd8aeda7ebbf975"
format_stamp: "Formatted at 2026-09-29 02:11:16 on dist-test-slave-q06p"
server_key: "062a1ed93247690f8690e7a3936f61ec"
server_key_iv: "bc5a001567eb82f1f4cab502859b9001"
server_key_version: "encryptionkey@0"
I20260929 02:11:16.297566 567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "20698ad91b244d8d9bd8aeda7ebbf975"
format_stamp: "Formatted at 2026-09-29 02:11:16 on dist-test-slave-q06p"
server_key: "062a1ed93247690f8690e7a3936f61ec"
server_key_iv: "bc5a001567eb82f1f4cab502859b9001"
server_key_version: "encryptionkey@0"
I20260929 02:11:16.302743 567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:11:16.306279 582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:16.307715 567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:11:16.307904 567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "20698ad91b244d8d9bd8aeda7ebbf975"
format_stamp: "Formatted at 2026-09-29 02:11:16 on dist-test-slave-q06p"
server_key: "062a1ed93247690f8690e7a3936f61ec"
server_key_iv: "bc5a001567eb82f1f4cab502859b9001"
server_key_version: "encryptionkey@0"
I20260929 02:11:16.308102 567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:16.333694 567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:16.338290 567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:16.338598 567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:16.358067 567 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38783
I20260929 02:11:16.358093 633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38783 every 8 connection(s)
I20260929 02:11:16.359580 567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:16.363410 634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:16.366403 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 567
I20260929 02:11:16.366585 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:16.367085 11995 external_mini_cluster.cc:1428] Setting key 2c0034f3186d4325acbacd89b9454bc6
I20260929 02:11:16.375614 634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975: Bootstrap starting.
I20260929 02:11:16.379588 634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:16.380964 634 log.cc:824] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:11:16 dist-test-slave-q06p krb5kdc[554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:11:16.384693 634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975: No bootstrap required, opened a new log
I20260929 02:11:16.390573 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:16.368991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:56011 (local address 127.0.0.1:38783)
0929 02:11:16.369741 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:16.369752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:16.369792 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:11:16.371475 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:16.371501 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:16.371902 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:16.372228 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:16.373906 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:16.375297 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:16.377150 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:16.377869 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:16.378017 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:16.383321 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:16.383328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:16.383343 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:16.383387 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:16.386559 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:16.387431 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:16.387438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:16.387443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:16.387613 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:16.388070 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:16.388074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:16.388076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:16.388572 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:16.388935 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:16.389309 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 02:11:16.389622 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":355,"threads_started":1}
I20260929 02:11:16.392961 634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20698ad91b244d8d9bd8aeda7ebbf975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } }
I20260929 02:11:16.393388 634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:16.393488 634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20698ad91b244d8d9bd8aeda7ebbf975, State: Initialized, Role: FOLLOWER
I20260929 02:11:16.394105 634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20698ad91b244d8d9bd8aeda7ebbf975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } }
I20260929 02:11:16.394397 634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:16.394564 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:16.394550 634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:16.394723 634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:16.396804 634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20698ad91b244d8d9bd8aeda7ebbf975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } }
I20260929 02:11:16.397363 634 leader_election.cc:302] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20698ad91b244d8d9bd8aeda7ebbf975; no voters:
I20260929 02:11:16.398041 634 leader_election.cc:288] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:16.398154 643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:16.400170 634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:16.401669 643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [term 1 LEADER]: Becoming Leader. State: Replica: 20698ad91b244d8d9bd8aeda7ebbf975, State: Running, Role: LEADER
I20260929 02:11:16.403607 643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20698ad91b244d8d9bd8aeda7ebbf975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } }
I20260929 02:11:16.406625 645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20698ad91b244d8d9bd8aeda7ebbf975. Latest consensus state: current_term: 1 leader_uuid: "20698ad91b244d8d9bd8aeda7ebbf975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20698ad91b244d8d9bd8aeda7ebbf975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } } }
I20260929 02:11:16.407049 645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:16.407572 644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20698ad91b244d8d9bd8aeda7ebbf975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20698ad91b244d8d9bd8aeda7ebbf975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } } }
I20260929 02:11:16.407858 644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:16.411583 657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:16.417237 657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:16.422572 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:16.399580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:57383 (local address 127.0.0.1:38783)
0929 02:11:16.399766 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:16.399771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:16.401589 (+ 1818us) server_negotiation.cc:407] Connection header received
0929 02:11:16.401879 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:16.401886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:16.402001 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:16.402139 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:16.404049 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:16.405578 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:16.408343 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:16.409419 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:16.409597 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:16.414017 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:16.414023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:16.414028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:16.414065 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:16.418203 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:16.419348 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:16.419358 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:16.419363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:16.419510 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:16.420115 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:16.420121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:16.420123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:16.420525 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:16.420802 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:16.422057 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0929 02:11:16.422224 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:11:16.424510 662 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:11:16.426991 657 catalog_manager.cc:1347] Generated new cluster ID: 76307512e03542e89b05c1e917229ad7
I20260929 02:11:16.427098 657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:16.459017 657 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:16.460451 657 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:16.470364 657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20698ad91b244d8d9bd8aeda7ebbf975: Generated new TSK 0
I20260929 02:11:16.471330 657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:16.480530 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 567
I20260929 02:11:16.481711 629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:16.576462 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 567
W20260929 02:11:16.593662 552 connection.cc:535] client connection to 127.0.0.1:38783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38783): Connection reset by peer (error 104)
2026-09-29T02:11:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:16 dist-test-slave-q06p krb5kdc[667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:16 dist-test-slave-q06p krb5kdc[667](info): set up 2 sockets
Sep 29 02:11:16 dist-test-slave-q06p krb5kdc[667](info): commencing operation
krb5kdc: starting...
W20260929 02:11:17.707828 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:17 dist-test-slave-q06p krb5kdc[667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:17.960045 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:18.302930 680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:18.303257 680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:18.303352 680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:18.311038 680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:18.311172 680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:18.311223 680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:18.311285 680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:18.311328 680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:18.321545 680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:18.322937 680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:18.324575 680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:18.336632 685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:18.336588 686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:18.336666 688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:18.337984 680 server_base.cc:1034] running on GCE node
I20260929 02:11:18.338901 680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:18.340368 680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:18.341573 680 hybrid_clock.cc:648] HybridClock initialized: now 1790647878341522 us; error 42 us; skew 500 ppm
I20260929 02:11:18.341948 680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:18 dist-test-slave-q06p krb5kdc[667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:18.345268 680 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:18.346889 680 webserver.cc:460] Webserver started at http://127.0.0.1:43875/ using document root <none> and password file <none>
I20260929 02:11:18.347633 680 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:18.347770 680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:18.348099 680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:18.351318 680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "b34566e788d94bdea4e64a5d88e98105"
format_stamp: "Formatted at 2026-09-29 02:11:18 on dist-test-slave-q06p"
server_key: "3e4d21ce0e452264a8156ffe1a21e0f7"
server_key_iv: "9ccb6b55ecacc598a4370739015b49fc"
server_key_version: "encryptionkey@0"
I20260929 02:11:18.352085 680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "b34566e788d94bdea4e64a5d88e98105"
format_stamp: "Formatted at 2026-09-29 02:11:18 on dist-test-slave-q06p"
server_key: "3e4d21ce0e452264a8156ffe1a21e0f7"
server_key_iv: "9ccb6b55ecacc598a4370739015b49fc"
server_key_version: "encryptionkey@0"
I20260929 02:11:18.357908 680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:11:18.361581 695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:18.362999 680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:11:18.363178 680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "b34566e788d94bdea4e64a5d88e98105"
format_stamp: "Formatted at 2026-09-29 02:11:18 on dist-test-slave-q06p"
server_key: "3e4d21ce0e452264a8156ffe1a21e0f7"
server_key_iv: "9ccb6b55ecacc598a4370739015b49fc"
server_key_version: "encryptionkey@0"
I20260929 02:11:18.363370 680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:18.386948 680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:18.391227 680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:18.391522 680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:18.409963 680 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44737
I20260929 02:11:18.410033 747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44737 every 8 connection(s)
I20260929 02:11:18.411378 680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:18.415259 748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:18.416818 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 680
I20260929 02:11:18.417062 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:18.417631 11995 external_mini_cluster.cc:1428] Setting key 14670be4246f084e823f45d4300bcadd
I20260929 02:11:18.430267 748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105: Bootstrap starting.
I20260929 02:11:18.431553 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:18.419547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:59445 (local address 127.0.0.1:44737)
0929 02:11:18.420327 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:18.420338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:18.420383 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:11:18.421986 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:18.422007 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:18.422425 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:18.422774 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:18.423520 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:18.424769 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:18.426957 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:18.427662 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:18.427807 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:18.429272 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:18.429279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:18.429292 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:18.429814 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:18.429887 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:18.429909 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:18.430017 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:18.430325 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 02:11:18.430522 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":362,"threads_started":1}
I20260929 02:11:18.433771 748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:18.435017 748 log.cc:824] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:18.435249 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:18.438061 748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105: No bootstrap required, opened a new log
I20260929 02:11:18.447171 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:18.440660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:52915 (local address 127.0.0.1:44737)
0929 02:11:18.440854 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:18.440862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:18.441283 (+ 421us) server_negotiation.cc:407] Connection header received
0929 02:11:18.441427 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:18.441432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:18.441521 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:18.441634 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:18.442083 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:18.443232 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:18.444512 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:18.445208 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:18.445347 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:18.446376 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:18.446384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:18.446389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:18.446418 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:18.446467 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:18.446496 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:18.446623 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:18.446764 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 02:11:18.446819 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:11:18.448166 748 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34566e788d94bdea4e64a5d88e98105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44737 } }
I20260929 02:11:18.448570 748 raft_consensus.cc:374] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:18.448688 748 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34566e788d94bdea4e64a5d88e98105, State: Initialized, Role: FOLLOWER
I20260929 02:11:18.449220 748 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34566e788d94bdea4e64a5d88e98105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44737 } }
I20260929 02:11:18.449424 748 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:18.449519 748 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:18.449651 748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:18.452198 748 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34566e788d94bdea4e64a5d88e98105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44737 } }
I20260929 02:11:18.452632 748 leader_election.cc:302] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34566e788d94bdea4e64a5d88e98105; no voters:
I20260929 02:11:18.453171 748 leader_election.cc:288] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:18.453363 758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:18.454120 758 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [term 1 LEADER]: Becoming Leader. State: Replica: b34566e788d94bdea4e64a5d88e98105, State: Running, Role: LEADER
I20260929 02:11:18.454648 758 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34566e788d94bdea4e64a5d88e98105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44737 } }
I20260929 02:11:18.455310 748 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:18.457634 759 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b34566e788d94bdea4e64a5d88e98105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34566e788d94bdea4e64a5d88e98105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44737 } } }
I20260929 02:11:18.458180 759 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:18.460041 766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:18.457655 760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34566e788d94bdea4e64a5d88e98105. Latest consensus state: current_term: 1 leader_uuid: "b34566e788d94bdea4e64a5d88e98105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34566e788d94bdea4e64a5d88e98105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44737 } } }
I20260929 02:11:18.463299 760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:18.466913 766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:18.477068 766 catalog_manager.cc:1347] Generated new cluster ID: 4a9f3348ec96447bacf0b10bfc335cea
I20260929 02:11:18.477218 766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:18.508603 766 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:18.510484 766 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:18.530530 766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b34566e788d94bdea4e64a5d88e98105: Generated new TSK 0
I20260929 02:11:18.531564 766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:18.550613 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 680
W20260929 02:11:18.551986 743 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.221:59445
W20260929 02:11:18.552778 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:18.552917 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 680
W20260929 02:11:18.571130 665 connection.cc:535] client connection to 127.0.0.1:44737 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44737): Connection reset by peer (error 104)
2026-09-29T02:11:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:18 dist-test-slave-q06p krb5kdc[779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:18 dist-test-slave-q06p krb5kdc[779](info): set up 2 sockets
Sep 29 02:11:18 dist-test-slave-q06p krb5kdc[779](info): commencing operation
krb5kdc: starting...
W20260929 02:11:19.651732 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:19 dist-test-slave-q06p krb5kdc[779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:19.924026 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:20.258888 792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:20.259191 792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:20.259289 792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:20.266702 792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:20.266829 792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:20.267017 792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:20.267132 792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:20.267227 792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:20.277966 792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:20.279414 792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:20.280822 792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:20.293058 799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:20.293125 801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:20.293726 792 server_base.cc:1034] running on GCE node
W20260929 02:11:20.293865 798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:20.294726 792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:20.296092 792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:20.297319 792 hybrid_clock.cc:648] HybridClock initialized: now 1790647880297277 us; error 57 us; skew 500 ppm
I20260929 02:11:20.297675 792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:20 dist-test-slave-q06p krb5kdc[779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:20.305627 792 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:20.307364 792 webserver.cc:460] Webserver started at http://127.0.0.1:33751/ using document root <none> and password file <none>
I20260929 02:11:20.308101 792 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:20.308234 792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:20.308528 792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:20.311990 792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "eecfdf98908f4a23afb9fd542d945f61"
format_stamp: "Formatted at 2026-09-29 02:11:20 on dist-test-slave-q06p"
server_key: "aa862cc5d994525f54be6a917a1cb7c3"
server_key_iv: "386427d1fd2153dfde8f1193597ccc03"
server_key_version: "encryptionkey@0"
I20260929 02:11:20.312840 792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "eecfdf98908f4a23afb9fd542d945f61"
format_stamp: "Formatted at 2026-09-29 02:11:20 on dist-test-slave-q06p"
server_key: "aa862cc5d994525f54be6a917a1cb7c3"
server_key_iv: "386427d1fd2153dfde8f1193597ccc03"
server_key_version: "encryptionkey@0"
I20260929 02:11:20.318485 792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:11:20.322170 808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:20.323570 792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:11:20.323762 792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "eecfdf98908f4a23afb9fd542d945f61"
format_stamp: "Formatted at 2026-09-29 02:11:20 on dist-test-slave-q06p"
server_key: "aa862cc5d994525f54be6a917a1cb7c3"
server_key_iv: "386427d1fd2153dfde8f1193597ccc03"
server_key_version: "encryptionkey@0"
I20260929 02:11:20.323961 792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:20.329710 792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:20.334024 792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:20.334338 792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:20.352854 792 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33147
I20260929 02:11:20.352900 859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33147 every 8 connection(s)
I20260929 02:11:20.354269 792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:20.358070 860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:20.358649 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 792
I20260929 02:11:20.358868 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:20.359318 11995 external_mini_cluster.cc:1428] Setting key 80ac06eff3be78757e9440bb50369de9
I20260929 02:11:20.367400 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:20.360969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:54375 (local address 127.0.0.1:33147)
0929 02:11:20.361762 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:20.361772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:20.361820 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:11:20.364033 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:20.364055 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:20.364410 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:20.364748 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:20.365818 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:20.365824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:20.365836 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:20.365849 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:20.365903 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:20.365924 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:20.366090 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:20.366182 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 02:11:20.366384 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":364,"threads_started":1}
I20260929 02:11:20.371105 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:20.371433 860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61: Bootstrap starting.
I20260929 02:11:20.374357 860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:20.375488 860 log.cc:824] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:20.378087 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:20.375556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:51249 (local address 127.0.0.1:33147)
0929 02:11:20.375759 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:20.375764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:20.376114 (+ 350us) server_negotiation.cc:407] Connection header received
0929 02:11:20.376305 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:20.376310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:20.376365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:20.376468 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:20.377444 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:20.377448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:20.377451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:20.377464 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:20.377502 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:20.377516 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:20.377598 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:20.377749 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 02:11:20.377792 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:11:20.378718 860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61: No bootstrap required, opened a new log
I20260929 02:11:20.385977 860 raft_consensus.cc:348] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecfdf98908f4a23afb9fd542d945f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260929 02:11:20.386384 860 raft_consensus.cc:374] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:20.386490 860 raft_consensus.cc:729] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecfdf98908f4a23afb9fd542d945f61, State: Initialized, Role: FOLLOWER
I20260929 02:11:20.387116 860 consensus_queue.cc:260] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecfdf98908f4a23afb9fd542d945f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260929 02:11:20.387372 860 raft_consensus.cc:388] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:20.387514 860 raft_consensus.cc:482] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:20.387686 860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:20.389883 860 raft_consensus.cc:504] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecfdf98908f4a23afb9fd542d945f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260929 02:11:20.390407 860 leader_election.cc:302] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eecfdf98908f4a23afb9fd542d945f61; no voters:
I20260929 02:11:20.391119 860 leader_election.cc:288] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:20.391240 870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:20.392257 870 raft_consensus.cc:686] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [term 1 LEADER]: Becoming Leader. State: Replica: eecfdf98908f4a23afb9fd542d945f61, State: Running, Role: LEADER
I20260929 02:11:20.392859 870 consensus_queue.cc:237] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecfdf98908f4a23afb9fd542d945f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260929 02:11:20.393610 860 sys_catalog.cc:564] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:20.395799 871 sys_catalog.cc:455] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eecfdf98908f4a23afb9fd542d945f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecfdf98908f4a23afb9fd542d945f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } } }
I20260929 02:11:20.396361 871 sys_catalog.cc:458] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:20.398945 872 sys_catalog.cc:455] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eecfdf98908f4a23afb9fd542d945f61. Latest consensus state: current_term: 1 leader_uuid: "eecfdf98908f4a23afb9fd542d945f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecfdf98908f4a23afb9fd542d945f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } } }
I20260929 02:11:20.400733 872 sys_catalog.cc:458] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:20.402132 879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:20.406286 879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:20.414515 879 catalog_manager.cc:1347] Generated new cluster ID: b776c92906d242ab8f52a5679d0ce9af
I20260929 02:11:20.414635 879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:20.425594 879 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:20.425978 879 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:20.434361 879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eecfdf98908f4a23afb9fd542d945f61: Generated new TSK 0
I20260929 02:11:20.435257 879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:20.442960 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 792
W20260929 02:11:20.444278 855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.221:54375
W20260929 02:11:20.444993 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 792: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:20.445144 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 792
2026-09-29T02:11:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1890 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:20 dist-test-slave-q06p krb5kdc[891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:20 dist-test-slave-q06p krb5kdc[891](info): set up 2 sockets
Sep 29 02:11:20 dist-test-slave-q06p krb5kdc[891](info): commencing operation
krb5kdc: starting...
W20260929 02:11:21.572551 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:21 dist-test-slave-q06p krb5kdc[891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:21.838645 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:22.185873 904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:22.186196 904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:22.186369 904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:22.193910 904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:22.194041 904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:22.194166 904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:22.194278 904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:22.194356 904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:22.205260 904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:22.206718 904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:22.208191 904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:22.221082 909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:22.221258 910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:22.221275 912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:22.222077 904 server_base.cc:1034] running on GCE node
I20260929 02:11:22.222957 904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:22.224315 904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:22.225569 904 hybrid_clock.cc:648] HybridClock initialized: now 1790647882225489 us; error 42 us; skew 500 ppm
I20260929 02:11:22.225929 904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:22 dist-test-slave-q06p krb5kdc[891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:22.229815 904 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:22.231560 904 webserver.cc:460] Webserver started at http://127.0.0.1:39661/ using document root <none> and password file <none>
I20260929 02:11:22.232324 904 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:22.232455 904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:22.232779 904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:22.236123 904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "4eb321108be14613a797ebe3eccc0c1b"
format_stamp: "Formatted at 2026-09-29 02:11:22 on dist-test-slave-q06p"
server_key: "d16d480886e4b7c2674b8f1611188520"
server_key_iv: "882a184688487e6b5b244f35b4f0fa52"
server_key_version: "encryptionkey@0"
I20260929 02:11:22.237010 904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "4eb321108be14613a797ebe3eccc0c1b"
format_stamp: "Formatted at 2026-09-29 02:11:22 on dist-test-slave-q06p"
server_key: "d16d480886e4b7c2674b8f1611188520"
server_key_iv: "882a184688487e6b5b244f35b4f0fa52"
server_key_version: "encryptionkey@0"
I20260929 02:11:22.242663 904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:11:22.246300 919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:22.247722 904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:11:22.247907 904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "4eb321108be14613a797ebe3eccc0c1b"
format_stamp: "Formatted at 2026-09-29 02:11:22 on dist-test-slave-q06p"
server_key: "d16d480886e4b7c2674b8f1611188520"
server_key_iv: "882a184688487e6b5b244f35b4f0fa52"
server_key_version: "encryptionkey@0"
I20260929 02:11:22.248108 904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:22.272780 904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:22.277302 904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:22.277608 904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:22.296736 904 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45297
I20260929 02:11:22.296768 971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45297 every 8 connection(s)
I20260929 02:11:22.298509 904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:22.302234 972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:22.302914 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 904
I20260929 02:11:22.303102 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:22.303526 11995 external_mini_cluster.cc:1428] Setting key fb476222acce9de84d61a53c3b32af0a
I20260929 02:11:22.317145 972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b: Bootstrap starting.
Sep 29 02:11:22 dist-test-slave-q06p krb5kdc[891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:11:22.320564 972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:22.321996 972 log.cc:824] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:22.325609 972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b: No bootstrap required, opened a new log
I20260929 02:11:22.329317 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:22.305447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:57667 (local address 127.0.0.1:45297)
0929 02:11:22.306244 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:22.306259 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:22.306314 (+ 55us) server_negotiation.cc:407] Connection header received
0929 02:11:22.308968 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:22.309000 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:22.309593 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:22.310075 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:22.311517 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:22.313558 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:22.314902 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:22.316023 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:22.316219 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:22.320463 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:22.320475 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:22.320496 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:22.320555 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:22.324481 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:22.325267 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:22.325275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:22.325283 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:22.325481 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:22.325983 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:22.325987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:22.325990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:22.326720 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:22.327099 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:22.327537 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 02:11:22.327897 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":319,"threads_started":1}
I20260929 02:11:22.333428 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:22.334972 972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb321108be14613a797ebe3eccc0c1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45297 } }
I20260929 02:11:22.335327 972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:22.335425 972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb321108be14613a797ebe3eccc0c1b, State: Initialized, Role: FOLLOWER
I20260929 02:11:22.335986 972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb321108be14613a797ebe3eccc0c1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45297 } }
I20260929 02:11:22.336239 972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:22.336390 972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:22.336562 972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:22.339440 972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb321108be14613a797ebe3eccc0c1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45297 } }
I20260929 02:11:22.339958 972 leader_election.cc:302] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eb321108be14613a797ebe3eccc0c1b; no voters:
I20260929 02:11:22.340641 972 leader_election.cc:288] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:22.340729 982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:22.343411 972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:22.347172 982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [term 1 LEADER]: Becoming Leader. State: Replica: 4eb321108be14613a797ebe3eccc0c1b, State: Running, Role: LEADER
I20260929 02:11:22.348029 982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb321108be14613a797ebe3eccc0c1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45297 } }
I20260929 02:11:22.351390 983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eb321108be14613a797ebe3eccc0c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb321108be14613a797ebe3eccc0c1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45297 } } }
I20260929 02:11:22.351830 983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:22.352597 984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eb321108be14613a797ebe3eccc0c1b. Latest consensus state: current_term: 1 leader_uuid: "4eb321108be14613a797ebe3eccc0c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb321108be14613a797ebe3eccc0c1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45297 } } }
I20260929 02:11:22.352965 984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:22.357645 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:22.338156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:60803 (local address 127.0.0.1:45297)
0929 02:11:22.338352 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:22.338362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:22.338748 (+ 386us) server_negotiation.cc:407] Connection header received
0929 02:11:22.338962 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:22.338968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:22.339044 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:22.339145 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:22.340625 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:22.341909 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:22.343336 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:22.343922 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:22.344060 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:22.346980 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:22.346992 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:22.346999 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:22.347043 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:22.352192 (+ 5149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:22.353233 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:22.353239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:22.353242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:22.353391 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:22.354008 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:22.354013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:22.354017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:22.354522 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:22.356063 (+ 1541us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:22.356365 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:11:22.356803 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":62}
I20260929 02:11:22.359754 990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:22.364279 990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:22.373313 990 catalog_manager.cc:1347] Generated new cluster ID: 8f96ac61ff8247a4993e270550ca17db
I20260929 02:11:22.373454 990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:22.418154 990 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:22.420130 990 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:22.433372 990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eb321108be14613a797ebe3eccc0c1b: Generated new TSK 0
I20260929 02:11:22.434296 990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:22.448478 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 904
I20260929 02:11:22.449769 967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:22.547750 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 904
W20260929 02:11:22.565766 889 connection.cc:535] client connection to 127.0.0.1:45297 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45297): Connection reset by peer (error 104)
2026-09-29T02:11:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2103 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:22 dist-test-slave-q06p krb5kdc[1004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:22 dist-test-slave-q06p krb5kdc[1004](info): set up 2 sockets
Sep 29 02:11:22 dist-test-slave-q06p krb5kdc[1004](info): commencing operation
krb5kdc: starting...
W20260929 02:11:23.679718 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:23 dist-test-slave-q06p krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:23.930375 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:24.251528 1017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:24.251861 1017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:24.252036 1017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:24.259169 1017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:24.259315 1017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:24.259418 1017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:24.259518 1017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:24.259605 1017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:24.269330 1017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42185
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:24.270599 1017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:24.271931 1017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:24.283704 1022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:24.283645 1023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:24.283731 1025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:24.284677 1017 server_base.cc:1034] running on GCE node
I20260929 02:11:24.285420 1017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:24.286672 1017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:24.287896 1017 hybrid_clock.cc:648] HybridClock initialized: now 1790647884287841 us; error 47 us; skew 500 ppm
I20260929 02:11:24.288260 1017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:24 dist-test-slave-q06p krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:24.291307 1017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:24.292747 1017 webserver.cc:460] Webserver started at http://127.0.0.1:40643/ using document root <none> and password file <none>
I20260929 02:11:24.293394 1017 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:24.293534 1017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:24.293829 1017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:24.296748 1017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "f41a7fd72e4c441ab4b7e01887ca0516"
format_stamp: "Formatted at 2026-09-29 02:11:24 on dist-test-slave-q06p"
server_key: "a3e40186a42540d04493dd0d979af470"
server_key_iv: "77d38816793ea3f14c13e4d4d04b48e3"
server_key_version: "encryptionkey@0"
I20260929 02:11:24.297477 1017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "f41a7fd72e4c441ab4b7e01887ca0516"
format_stamp: "Formatted at 2026-09-29 02:11:24 on dist-test-slave-q06p"
server_key: "a3e40186a42540d04493dd0d979af470"
server_key_iv: "77d38816793ea3f14c13e4d4d04b48e3"
server_key_version: "encryptionkey@0"
I20260929 02:11:24.302304 1017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 02:11:24.305567 1032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:24.306800 1017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:11:24.307011 1017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "f41a7fd72e4c441ab4b7e01887ca0516"
format_stamp: "Formatted at 2026-09-29 02:11:24 on dist-test-slave-q06p"
server_key: "a3e40186a42540d04493dd0d979af470"
server_key_iv: "77d38816793ea3f14c13e4d4d04b48e3"
server_key_version: "encryptionkey@0"
I20260929 02:11:24.307195 1017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:24.349468 1017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:24.353845 1017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:24.354151 1017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:24.372720 1017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42185
I20260929 02:11:24.372757 1084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42185 every 8 connection(s)
I20260929 02:11:24.374058 1017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:24.376363 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1017
I20260929 02:11:24.376540 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:24.377074 11995 external_mini_cluster.cc:1428] Setting key 89ce2bac8e0f6afa6eb9f727bdb0de5a
I20260929 02:11:24.378386 1085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:24.389904 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:24.378908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:59425 (local address 127.0.0.1:42185)
0929 02:11:24.379642 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:24.379652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:24.379691 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:11:24.381239 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:24.381258 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:24.381635 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:24.381964 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:24.382527 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:24.383901 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:24.385693 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:24.386317 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:24.386434 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:24.387722 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:24.387729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:24.387741 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:24.388263 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:24.388334 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:24.388357 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:24.388476 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:24.388757 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 02:11:24.388959 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":337,"threads_started":1}
I20260929 02:11:24.391659 1085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516: Bootstrap starting.
I20260929 02:11:24.393716 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:24.395037 1085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:24.396055 1085 log.cc:824] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:24.399467 1085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516: No bootstrap required, opened a new log
I20260929 02:11:24.404814 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:24.397949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:55771 (local address 127.0.0.1:42185)
0929 02:11:24.398158 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:24.398165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:24.398491 (+ 326us) server_negotiation.cc:407] Connection header received
0929 02:11:24.398681 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:24.398685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:24.398742 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:24.398820 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:24.399468 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:24.400277 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:24.402161 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:24.402624 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:24.402716 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:24.403931 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:24.403935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:24.403939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:24.403954 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:24.403989 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:24.404003 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:24.404091 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:24.404294 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 02:11:24.404463 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:11:24.408422 1085 raft_consensus.cc:348] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41a7fd72e4c441ab4b7e01887ca0516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } }
I20260929 02:11:24.408797 1085 raft_consensus.cc:374] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:24.408907 1085 raft_consensus.cc:729] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41a7fd72e4c441ab4b7e01887ca0516, State: Initialized, Role: FOLLOWER
I20260929 02:11:24.409507 1085 consensus_queue.cc:260] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41a7fd72e4c441ab4b7e01887ca0516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } }
I20260929 02:11:24.409744 1085 raft_consensus.cc:388] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:24.409907 1085 raft_consensus.cc:482] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:24.410076 1085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:24.412539 1085 raft_consensus.cc:504] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41a7fd72e4c441ab4b7e01887ca0516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } }
I20260929 02:11:24.413056 1085 leader_election.cc:302] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f41a7fd72e4c441ab4b7e01887ca0516; no voters:
I20260929 02:11:24.413651 1085 leader_election.cc:288] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:24.413810 1095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:24.414466 1095 raft_consensus.cc:686] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [term 1 LEADER]: Becoming Leader. State: Replica: f41a7fd72e4c441ab4b7e01887ca0516, State: Running, Role: LEADER
I20260929 02:11:24.415014 1095 consensus_queue.cc:237] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41a7fd72e4c441ab4b7e01887ca0516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } }
I20260929 02:11:24.415706 1085 sys_catalog.cc:564] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:24.417838 1097 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41a7fd72e4c441ab4b7e01887ca0516. Latest consensus state: current_term: 1 leader_uuid: "f41a7fd72e4c441ab4b7e01887ca0516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41a7fd72e4c441ab4b7e01887ca0516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } } }
I20260929 02:11:24.418187 1097 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:24.418478 1096 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f41a7fd72e4c441ab4b7e01887ca0516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41a7fd72e4c441ab4b7e01887ca0516" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } } }
I20260929 02:11:24.418974 1096 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:24.420819 1100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:24.425500 1100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:24.435732 1100 catalog_manager.cc:1347] Generated new cluster ID: ef7eaabcc7274d6da1cc97a0c82ee796
I20260929 02:11:24.435858 1100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:24.446594 1100 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:24.448403 1100 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:24.457657 1100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f41a7fd72e4c441ab4b7e01887ca0516: Generated new TSK 0
I20260929 02:11:24.458508 1100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:24.474457 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1017
W20260929 02:11:24.476379 1080 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.221:59425
W20260929 02:11:24.477103 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1017: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:24.477247 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1017
W20260929 02:11:24.494081 1002 connection.cc:535] client connection to 127.0.0.1:42185 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42185): Connection reset by peer (error 104)
2026-09-29T02:11:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:24 dist-test-slave-q06p krb5kdc[1116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:24 dist-test-slave-q06p krb5kdc[1116](info): set up 2 sockets
Sep 29 02:11:24 dist-test-slave-q06p krb5kdc[1116](info): commencing operation
krb5kdc: starting...
W20260929 02:11:25.579841 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:25 dist-test-slave-q06p krb5kdc[1116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:11:25.833884 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:26.154048 1129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:26.154362 1129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:26.154541 1129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:26.161384 1129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:26.161520 1129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:26.161629 1129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:26.161738 1129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:26.161821 1129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:26.171763 1129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:26.172967 1129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:26.174244 1129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:26.185609 1134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:26.185704 1137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:26.185681 1135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:26.186599 1129 server_base.cc:1034] running on GCE node
I20260929 02:11:26.187412 1129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:26.188710 1129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:26.189954 1129 hybrid_clock.cc:648] HybridClock initialized: now 1790647886189906 us; error 37 us; skew 500 ppm
I20260929 02:11:26.190299 1129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:26 dist-test-slave-q06p krb5kdc[1116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:26.199638 1129 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:11:26.201200 1129 webserver.cc:460] Webserver started at http://127.0.0.1:34753/ using document root <none> and password file <none>
I20260929 02:11:26.201905 1129 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:26.202055 1129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:26.202382 1129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:26.205535 1129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "bd840ca1f20c46fd94a3c74269014688"
format_stamp: "Formatted at 2026-09-29 02:11:26 on dist-test-slave-q06p"
server_key: "d49bb9610f855339976a5a97734b0a82"
server_key_iv: "3cd62aed276a54f5fe45f63b6e01ccfe"
server_key_version: "encryptionkey@0"
I20260929 02:11:26.206303 1129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "bd840ca1f20c46fd94a3c74269014688"
format_stamp: "Formatted at 2026-09-29 02:11:26 on dist-test-slave-q06p"
server_key: "d49bb9610f855339976a5a97734b0a82"
server_key_iv: "3cd62aed276a54f5fe45f63b6e01ccfe"
server_key_version: "encryptionkey@0"
I20260929 02:11:26.211490 1129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 02:11:26.214807 1144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:26.216226 1129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:11:26.216399 1129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "bd840ca1f20c46fd94a3c74269014688"
format_stamp: "Formatted at 2026-09-29 02:11:26 on dist-test-slave-q06p"
server_key: "d49bb9610f855339976a5a97734b0a82"
server_key_iv: "3cd62aed276a54f5fe45f63b6e01ccfe"
server_key_version: "encryptionkey@0"
I20260929 02:11:26.216588 1129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:26.222113 1129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:26.225960 1129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:26.226277 1129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:26.244692 1129 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45203
I20260929 02:11:26.244738 1195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45203 every 8 connection(s)
I20260929 02:11:26.246181 1129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:26.246948 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1129
I20260929 02:11:26.247146 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:26.247707 11995 external_mini_cluster.cc:1428] Setting key feb1934b25af7913bd4070bd596120a8
I20260929 02:11:26.250631 1196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:26.255395 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:26.249511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:44321 (local address 127.0.0.1:45203)
0929 02:11:26.250259 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:26.250271 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:26.250317 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:11:26.252098 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:26.252119 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:26.252429 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:26.252798 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:26.253794 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:26.253800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:26.253812 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:26.253825 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:26.253888 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:26.253906 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:26.254170 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:26.254306 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 02:11:26.254468 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":311,"threads_started":1}
I20260929 02:11:26.259073 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:26.262074 1196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688: Bootstrap starting.
I20260929 02:11:26.265656 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:26.263354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:36809 (local address 127.0.0.1:45203)
0929 02:11:26.263541 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:26.263549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:26.263807 (+ 258us) server_negotiation.cc:407] Connection header received
0929 02:11:26.263948 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:26.263954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:26.264027 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:26.264153 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:26.265007 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:26.265012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:26.265016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:26.265033 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:26.265076 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:26.265095 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:26.265215 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:26.265281 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 02:11:26.265335 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:11:26.265751 1196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:26.266815 1196 log.cc:824] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:26.269459 1196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688: No bootstrap required, opened a new log
I20260929 02:11:26.276707 1196 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd840ca1f20c46fd94a3c74269014688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45203 } }
I20260929 02:11:26.277027 1196 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:26.277096 1196 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd840ca1f20c46fd94a3c74269014688, State: Initialized, Role: FOLLOWER
I20260929 02:11:26.277678 1196 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd840ca1f20c46fd94a3c74269014688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45203 } }
I20260929 02:11:26.277944 1196 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:26.278088 1196 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:26.278296 1196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:26.280462 1196 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd840ca1f20c46fd94a3c74269014688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45203 } }
I20260929 02:11:26.280874 1196 leader_election.cc:302] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd840ca1f20c46fd94a3c74269014688; no voters:
I20260929 02:11:26.281353 1196 leader_election.cc:288] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:26.281533 1206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:26.282231 1206 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [term 1 LEADER]: Becoming Leader. State: Replica: bd840ca1f20c46fd94a3c74269014688, State: Running, Role: LEADER
I20260929 02:11:26.282768 1206 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd840ca1f20c46fd94a3c74269014688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45203 } }
I20260929 02:11:26.283617 1196 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:26.286175 1207 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd840ca1f20c46fd94a3c74269014688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd840ca1f20c46fd94a3c74269014688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45203 } } }
I20260929 02:11:26.286659 1207 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:26.288184 1213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:26.288089 1208 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd840ca1f20c46fd94a3c74269014688. Latest consensus state: current_term: 1 leader_uuid: "bd840ca1f20c46fd94a3c74269014688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd840ca1f20c46fd94a3c74269014688" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45203 } } }
I20260929 02:11:26.288401 1208 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:26.293529 1213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:26.301688 1213 catalog_manager.cc:1347] Generated new cluster ID: 9d9ea96ce6de4bceb94ffbbe6e95629d
I20260929 02:11:26.301795 1213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:26.312161 1213 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:26.312577 1213 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:26.320129 1213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd840ca1f20c46fd94a3c74269014688: Generated new TSK 0
I20260929 02:11:26.321064 1213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:26.337668 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1129
W20260929 02:11:26.339036 1191 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.221:44321
W20260929 02:11:26.339803 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1129: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:26.339956 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1129
2026-09-29T02:11:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1862 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:26 dist-test-slave-q06p krb5kdc[1228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:26 dist-test-slave-q06p krb5kdc[1228](info): set up 2 sockets
Sep 29 02:11:26 dist-test-slave-q06p krb5kdc[1228](info): commencing operation
krb5kdc: starting...
W20260929 02:11:27.427645 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:27 dist-test-slave-q06p krb5kdc[1228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:11:27.684247 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34909
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:34909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:28.004004 1241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:28.004372 1241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:28.004576 1241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:28.011745 1241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:28.011899 1241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:28.012008 1241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:28.012104 1241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:28.012195 1241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:28.021957 1241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:34909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:28.023293 1241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:28.024605 1241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:28.036543 1246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:28.036631 1247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:28.036631 1249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:28.037410 1241 server_base.cc:1034] running on GCE node
I20260929 02:11:28.038261 1241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:28.039723 1241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:28.040927 1241 hybrid_clock.cc:648] HybridClock initialized: now 1790647888040876 us; error 36 us; skew 500 ppm
I20260929 02:11:28.041276 1241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:28 dist-test-slave-q06p krb5kdc[1228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647888, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:28.044417 1241 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:11:28.045895 1241 webserver.cc:460] Webserver started at http://127.11.182.254:33729/ using document root <none> and password file <none>
I20260929 02:11:28.046775 1241 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:28.047043 1241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:28.047403 1241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:28.050685 1241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "a8d20fad88e74a779e8f7d6c23c8dde6"
format_stamp: "Formatted at 2026-09-29 02:11:28 on dist-test-slave-q06p"
server_key: "80b7944e8d01b1531d3973616594e3fb"
server_key_iv: "0ddc61fa97537cb0cd5471c68efd9113"
server_key_version: "encryptionkey@0"
I20260929 02:11:28.051517 1241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "a8d20fad88e74a779e8f7d6c23c8dde6"
format_stamp: "Formatted at 2026-09-29 02:11:28 on dist-test-slave-q06p"
server_key: "80b7944e8d01b1531d3973616594e3fb"
server_key_iv: "0ddc61fa97537cb0cd5471c68efd9113"
server_key_version: "encryptionkey@0"
I20260929 02:11:28.056762 1241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:11:28.060120 1256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:28.061334 1241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:11:28.061511 1241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "a8d20fad88e74a779e8f7d6c23c8dde6"
format_stamp: "Formatted at 2026-09-29 02:11:28 on dist-test-slave-q06p"
server_key: "80b7944e8d01b1531d3973616594e3fb"
server_key_iv: "0ddc61fa97537cb0cd5471c68efd9113"
server_key_version: "encryptionkey@0"
I20260929 02:11:28.061674 1241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:28.082130 1241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:28.085903 1241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:28.086185 1241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:28.104241 1241 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:34909
I20260929 02:11:28.104274 1308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:34909 every 8 connection(s)
I20260929 02:11:28.105585 1241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:28.107817 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1241
I20260929 02:11:28.108000 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:28.108455 11995 external_mini_cluster.cc:1428] Setting key aa9dbe64a72b9b793713594b4fbec9d1
I20260929 02:11:28.109643 1309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:11:28 dist-test-slave-q06p krb5kdc[1228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:11:28.122920 1309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6: Bootstrap starting.
I20260929 02:11:28.127103 1309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:28.128396 1309 log.cc:824] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:28.132015 1309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6: No bootstrap required, opened a new log
I20260929 02:11:28.132355 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:28.110224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47401 (local address 127.11.182.254:34909)
0929 02:11:28.110998 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:28.111012 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:28.111062 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:11:28.112905 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:28.112932 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:28.113445 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:28.113859 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:28.115213 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:28.117238 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:28.118409 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:28.119455 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:28.119634 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:28.123556 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:28.123566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:28.123583 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:28.123631 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:28.127703 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:28.128555 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:28.128564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:28.128572 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:28.128773 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:28.129346 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:28.129349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:28.129351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:28.129973 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:28.130298 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:28.130670 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 02:11:28.131031 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":531,"thread_start_us":304,"threads_started":1}
I20260929 02:11:28.136422 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:28.140650 1309 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d20fad88e74a779e8f7d6c23c8dde6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34909 } }
I20260929 02:11:28.141229 1309 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:28.141373 1309 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d20fad88e74a779e8f7d6c23c8dde6, State: Initialized, Role: FOLLOWER
I20260929 02:11:28.142037 1309 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d20fad88e74a779e8f7d6c23c8dde6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34909 } }
I20260929 02:11:28.142277 1309 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:28.142418 1309 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:28.142596 1309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:28.144889 1309 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d20fad88e74a779e8f7d6c23c8dde6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34909 } }
I20260929 02:11:28.145382 1309 leader_election.cc:302] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8d20fad88e74a779e8f7d6c23c8dde6; no voters:
I20260929 02:11:28.146051 1309 leader_election.cc:288] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:28.146173 1319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:28.146898 1319 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [term 1 LEADER]: Becoming Leader. State: Replica: a8d20fad88e74a779e8f7d6c23c8dde6, State: Running, Role: LEADER
I20260929 02:11:28.147655 1319 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d20fad88e74a779e8f7d6c23c8dde6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34909 } }
I20260929 02:11:28.149968 1320 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8d20fad88e74a779e8f7d6c23c8dde6. Latest consensus state: current_term: 1 leader_uuid: "a8d20fad88e74a779e8f7d6c23c8dde6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d20fad88e74a779e8f7d6c23c8dde6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34909 } } }
I20260929 02:11:28.150391 1320 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:28.150873 1309 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:28.150738 1321 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8d20fad88e74a779e8f7d6c23c8dde6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d20fad88e74a779e8f7d6c23c8dde6" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34909 } } }
I20260929 02:11:28.151158 1321 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:28.152091 1323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:28.159607 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:28.140853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41919 (local address 127.11.182.254:34909)
0929 02:11:28.141060 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:28.141067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:28.141476 (+ 409us) server_negotiation.cc:407] Connection header received
0929 02:11:28.141676 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:28.141681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:28.141760 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:28.141862 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:28.143512 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:28.144294 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:28.145536 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:28.146110 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:28.146249 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:28.148852 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:28.148861 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:28.148868 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:28.148916 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:28.154343 (+ 5427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:28.155631 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:28.155639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:28.155645 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:28.155869 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:28.156572 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:28.156579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:28.156583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:28.157024 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:28.158532 (+ 1508us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:28.158785 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 02:11:28.159041 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:11:28.167984 1323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:28.177030 1323 catalog_manager.cc:1347] Generated new cluster ID: 6d1c4232d01a4aac984c7e6103169bc3
I20260929 02:11:28.177148 1323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:28.194643 1323 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:28.195928 1323 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:28.205411 1323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8d20fad88e74a779e8f7d6c23c8dde6: Generated new TSK 0
I20260929 02:11:28.206207 1323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:28.212045 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1241
I20260929 02:11:28.213135 1304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:28.302678 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1241
W20260929 02:11:28.321367 1226 connection.cc:535] client connection to 127.11.182.254:34909 recv error: Network error: failed to read from TLS socket (remote: 127.11.182.254:34909): Connection reset by peer (error 104)
2026-09-29T02:11:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:28 dist-test-slave-q06p krb5kdc[1341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:28 dist-test-slave-q06p krb5kdc[1341](info): set up 2 sockets
Sep 29 02:11:28 dist-test-slave-q06p krb5kdc[1341](info): commencing operation
krb5kdc: starting...
W20260929 02:11:29.419697 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:29 dist-test-slave-q06p krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:11:29.678012 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34665
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:34665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:30.002090 1354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:30.002444 1354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:30.002604 1354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:30.009833 1354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:30.009991 1354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:30.010102 1354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:30.010216 1354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:30.010301 1354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:30.020272 1354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:34665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:34665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:30.021581 1354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:30.022889 1354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:30.034951 1359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:30.034911 1360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:30.034888 1362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:30.035892 1354 server_base.cc:1034] running on GCE node
I20260929 02:11:30.036659 1354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:30.038069 1354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:30.039294 1354 hybrid_clock.cc:648] HybridClock initialized: now 1790647890039260 us; error 40 us; skew 500 ppm
I20260929 02:11:30.039639 1354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:30 dist-test-slave-q06p krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647890, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:30.042593 1354 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:11:30.044070 1354 webserver.cc:460] Webserver started at http://127.11.182.254:39827/ using document root <none> and password file <none>
I20260929 02:11:30.044750 1354 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:30.044891 1354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:30.045193 1354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:30.048167 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee"
format_stamp: "Formatted at 2026-09-29 02:11:30 on dist-test-slave-q06p"
server_key: "0f05eb276ab2842b52bc07c73ca3f275"
server_key_iv: "a58b9ce5257b3d97e032dc950cd4bdcf"
server_key_version: "encryptionkey@0"
I20260929 02:11:30.048962 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee"
format_stamp: "Formatted at 2026-09-29 02:11:30 on dist-test-slave-q06p"
server_key: "0f05eb276ab2842b52bc07c73ca3f275"
server_key_iv: "a58b9ce5257b3d97e032dc950cd4bdcf"
server_key_version: "encryptionkey@0"
I20260929 02:11:30.054138 1354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:11:30.057500 1370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:30.058784 1354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:11:30.058984 1354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee"
format_stamp: "Formatted at 2026-09-29 02:11:30 on dist-test-slave-q06p"
server_key: "0f05eb276ab2842b52bc07c73ca3f275"
server_key_iv: "a58b9ce5257b3d97e032dc950cd4bdcf"
server_key_version: "encryptionkey@0"
I20260929 02:11:30.059180 1354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:30.069416 1354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:30.073298 1354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:30.073561 1354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:30.092427 1354 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:34665
I20260929 02:11:30.092447 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:34665 every 8 connection(s)
I20260929 02:11:30.093834 1354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:30.097684 1422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:30.102203 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1354
I20260929 02:11:30.102399 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:30.102823 11995 external_mini_cluster.cc:1428] Setting key 252fc10d4098ae0178962ded1689d85f
I20260929 02:11:30.109182 1422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee: Bootstrap starting.
I20260929 02:11:30.112983 1422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:30.114351 1422 log.cc:824] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:30.116474 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:30.104611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51871 (local address 127.11.182.254:34665)
0929 02:11:30.105367 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:30.105377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:30.105414 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:11:30.107517 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:30.107538 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:30.107892 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:30.108184 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:30.108817 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:30.110160 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:30.112125 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:30.112738 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:30.112861 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:30.114243 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:30.114250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:30.114264 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:30.114840 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:30.114985 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:30.115009 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:30.115185 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:30.115446 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 02:11:30.115605 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":332,"threads_started":1}
I20260929 02:11:30.117476 1422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee: No bootstrap required, opened a new log
I20260929 02:11:30.120122 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:30.126370 1422 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34665 } }
I20260929 02:11:30.126777 1422 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:30.126912 1422 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f79a0a4fa24aa0b8fad82971ff63ee, State: Initialized, Role: FOLLOWER
I20260929 02:11:30.127537 1422 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34665 } }
I20260929 02:11:30.127823 1422 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:30.127990 1422 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:30.128211 1422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:30.130776 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:30.124288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52021 (local address 127.11.182.254:34665)
0929 02:11:30.124438 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:30.124442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:30.124917 (+ 475us) server_negotiation.cc:407] Connection header received
0929 02:11:30.125130 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:30.125139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:30.125198 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:30.125310 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:30.125781 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:30.126507 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:30.128168 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:30.128686 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:30.128834 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:30.130055 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:30.130060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:30.130063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:30.130082 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:30.130120 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:30.130137 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:30.130259 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:30.130445 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 02:11:30.130500 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:11:30.131261 1422 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34665 } }
I20260929 02:11:30.131855 1422 leader_election.cc:302] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8f79a0a4fa24aa0b8fad82971ff63ee; no voters:
I20260929 02:11:30.132591 1422 leader_election.cc:288] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:30.132644 1432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:30.133322 1432 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [term 1 LEADER]: Becoming Leader. State: Replica: d8f79a0a4fa24aa0b8fad82971ff63ee, State: Running, Role: LEADER
I20260929 02:11:30.133960 1432 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34665 } }
I20260929 02:11:30.135075 1422 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:30.136911 1433 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8f79a0a4fa24aa0b8fad82971ff63ee. Latest consensus state: current_term: 1 leader_uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34665 } } }
I20260929 02:11:30.139240 1433 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:30.137409 1434 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f79a0a4fa24aa0b8fad82971ff63ee" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 34665 } } }
I20260929 02:11:30.139453 1434 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:30.141043 1442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:30.146281 1442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:30.155635 1442 catalog_manager.cc:1347] Generated new cluster ID: e943df18b3fe41afb03a51c8c1cf59ec
I20260929 02:11:30.155766 1442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:30.167528 1442 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:30.168882 1442 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:30.176225 1442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8f79a0a4fa24aa0b8fad82971ff63ee: Generated new TSK 0
I20260929 02:11:30.177016 1442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:30.184116 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1354
W20260929 02:11:30.185277 1417 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51871
W20260929 02:11:30.186038 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:30.186178 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1354
W20260929 02:11:30.199918 1339 connection.cc:535] client connection to 127.11.182.254:34665 recv error: Network error: failed to read from TLS socket (remote: 127.11.182.254:34665): Connection reset by peer (error 104)
2026-09-29T02:11:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:30 dist-test-slave-q06p krb5kdc[1453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:30 dist-test-slave-q06p krb5kdc[1453](info): set up 2 sockets
Sep 29 02:11:30 dist-test-slave-q06p krb5kdc[1453](info): commencing operation
krb5kdc: starting...
W20260929 02:11:31.295791 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:31 dist-test-slave-q06p krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:11:31.539091 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33045
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:33045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:31.868119 1466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:31.868453 1466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:31.868660 1466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:31.875900 1466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:31.876063 1466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:31.876173 1466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:31.876284 1466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:31.876360 1466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:31.886567 1466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:33045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:33045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:31.887918 1466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:31.889240 1466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:31.901594 1472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:31.901624 1474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:31.901664 1471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:31.902933 1466 server_base.cc:1034] running on GCE node
I20260929 02:11:31.903759 1466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:31.905146 1466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:31.906375 1466 hybrid_clock.cc:648] HybridClock initialized: now 1790647891906339 us; error 62 us; skew 500 ppm
I20260929 02:11:31.906733 1466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:31 dist-test-slave-q06p krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647891, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:31.916144 1466 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:11:31.917641 1466 webserver.cc:460] Webserver started at http://127.11.182.254:40401/ using document root <none> and password file <none>
I20260929 02:11:31.918324 1466 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:31.918469 1466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:31.918772 1466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:31.921657 1466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "0dbe329a7b514dc3aa1062216432e29d"
format_stamp: "Formatted at 2026-09-29 02:11:31 on dist-test-slave-q06p"
server_key: "4737cff18e7287d35a3bb4074aa80abf"
server_key_iv: "89f2787f1522c497a1ed8d1b7962a3a2"
server_key_version: "encryptionkey@0"
I20260929 02:11:31.922495 1466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "0dbe329a7b514dc3aa1062216432e29d"
format_stamp: "Formatted at 2026-09-29 02:11:31 on dist-test-slave-q06p"
server_key: "4737cff18e7287d35a3bb4074aa80abf"
server_key_iv: "89f2787f1522c497a1ed8d1b7962a3a2"
server_key_version: "encryptionkey@0"
I20260929 02:11:31.927585 1466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 02:11:31.931006 1481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:31.932348 1466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:11:31.932535 1466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "0dbe329a7b514dc3aa1062216432e29d"
format_stamp: "Formatted at 2026-09-29 02:11:31 on dist-test-slave-q06p"
server_key: "4737cff18e7287d35a3bb4074aa80abf"
server_key_iv: "89f2787f1522c497a1ed8d1b7962a3a2"
server_key_version: "encryptionkey@0"
I20260929 02:11:31.932760 1466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:31.938436 1466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:31.942636 1466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:31.942999 1466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:31.961133 1466 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:33045
I20260929 02:11:31.961236 1533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:33045 every 8 connection(s)
I20260929 02:11:31.962469 1466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:31.963419 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1466
I20260929 02:11:31.963608 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:31.964216 11995 external_mini_cluster.cc:1428] Setting key 6d1de5dba458adf970119e2d60822095
I20260929 02:11:31.967074 1534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:31.971705 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:31.965993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40675 (local address 127.11.182.254:33045)
0929 02:11:31.966739 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:31.966751 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:31.966788 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:11:31.968461 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:31.968482 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:31.968773 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:31.969140 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:31.970136 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:31.970142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:31.970154 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:31.970173 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:31.970233 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:31.970251 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:31.970491 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:31.970632 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 02:11:31.970792 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":331,"threads_started":1}
I20260929 02:11:31.975317 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:31.978482 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d: Bootstrap starting.
I20260929 02:11:31.982008 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:31.979571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33525 (local address 127.11.182.254:33045)
0929 02:11:31.979769 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:31.979777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:31.980122 (+ 345us) server_negotiation.cc:407] Connection header received
0929 02:11:31.980322 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:31.980327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:31.980383 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:31.980480 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:31.981357 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:31.981362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:31.981366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:31.981383 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:31.981429 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:31.981443 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:31.981554 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:31.981678 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 02:11:31.981736 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:11:31.982012 1534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:31.983464 1534 log.cc:824] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:31.986076 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d: No bootstrap required, opened a new log
I20260929 02:11:31.994467 1534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbe329a7b514dc3aa1062216432e29d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } }
I20260929 02:11:31.994879 1534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:31.995008 1534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbe329a7b514dc3aa1062216432e29d, State: Initialized, Role: FOLLOWER
I20260929 02:11:31.995575 1534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbe329a7b514dc3aa1062216432e29d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } }
I20260929 02:11:31.995864 1534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:31.996029 1534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:31.996234 1534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:31.998566 1534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbe329a7b514dc3aa1062216432e29d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } }
I20260929 02:11:31.999152 1534 leader_election.cc:302] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dbe329a7b514dc3aa1062216432e29d; no voters:
I20260929 02:11:31.999763 1534 leader_election.cc:288] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:31.999881 1544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:32.000577 1544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [term 1 LEADER]: Becoming Leader. State: Replica: 0dbe329a7b514dc3aa1062216432e29d, State: Running, Role: LEADER
I20260929 02:11:32.001135 1544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbe329a7b514dc3aa1062216432e29d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } }
I20260929 02:11:32.001979 1534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:32.003867 1546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dbe329a7b514dc3aa1062216432e29d. Latest consensus state: current_term: 1 leader_uuid: "0dbe329a7b514dc3aa1062216432e29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbe329a7b514dc3aa1062216432e29d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } } }
I20260929 02:11:32.003922 1545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dbe329a7b514dc3aa1062216432e29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbe329a7b514dc3aa1062216432e29d" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 33045 } } }
I20260929 02:11:32.004348 1545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:32.004348 1546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:32.005484 1551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:32.012053 1551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:32.020903 1551 catalog_manager.cc:1347] Generated new cluster ID: 2b0056dafaba4a899376956f627e8c7f
I20260929 02:11:32.021024 1551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:32.043502 1551 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:32.043965 1551 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:32.050993 1551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dbe329a7b514dc3aa1062216432e29d: Generated new TSK 0
I20260929 02:11:32.051997 1551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:32.071789 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1466
W20260929 02:11:32.073107 1529 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40675
W20260929 02:11:32.073843 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1466: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:32.073987 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1466
2026-09-29T02:11:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:32 dist-test-slave-q06p krb5kdc[1565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:32 dist-test-slave-q06p krb5kdc[1565](info): set up 2 sockets
Sep 29 02:11:32 dist-test-slave-q06p krb5kdc[1565](info): commencing operation
krb5kdc: starting...
W20260929 02:11:33.183907 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:33 dist-test-slave-q06p krb5kdc[1565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:11:33.426594 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40455
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:45001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:40455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:33.751171 1578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:33.751435 1578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:33.751626 1578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:33.758571 1578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:33.758697 1578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:33.758796 1578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:33.758919 1578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:33.759003 1578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:33.768553 1578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:45001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:40455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:40455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:33.769804 1578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:33.771168 1578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:33.783133 1583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:33.783176 1584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:33.783241 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:33.784056 1578 server_base.cc:1034] running on GCE node
I20260929 02:11:33.784888 1578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:33.786351 1578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:33.787545 1578 hybrid_clock.cc:648] HybridClock initialized: now 1790647893787490 us; error 49 us; skew 500 ppm
I20260929 02:11:33.787886 1578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:33 dist-test-slave-q06p krb5kdc[1565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647893, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:33.790948 1578 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:11:33.792277 1578 webserver.cc:460] Webserver started at http://127.11.182.254:35089/ using document root <none> and password file <none>
I20260929 02:11:33.792901 1578 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:33.793007 1578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:33.793252 1578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:33.796087 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "bf1c3adcbe684de8b00baaf01368325a"
format_stamp: "Formatted at 2026-09-29 02:11:33 on dist-test-slave-q06p"
server_key: "21156f1fe7f84e502bba2896e43bfd9d"
server_key_iv: "6d8b6de8872631da3136e93976ff8f02"
server_key_version: "encryptionkey@0"
I20260929 02:11:33.796793 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "bf1c3adcbe684de8b00baaf01368325a"
format_stamp: "Formatted at 2026-09-29 02:11:33 on dist-test-slave-q06p"
server_key: "21156f1fe7f84e502bba2896e43bfd9d"
server_key_iv: "6d8b6de8872631da3136e93976ff8f02"
server_key_version: "encryptionkey@0"
I20260929 02:11:33.801584 1578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:11:33.804872 1593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:33.806217 1578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:11:33.806389 1578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "bf1c3adcbe684de8b00baaf01368325a"
format_stamp: "Formatted at 2026-09-29 02:11:33 on dist-test-slave-q06p"
server_key: "21156f1fe7f84e502bba2896e43bfd9d"
server_key_iv: "6d8b6de8872631da3136e93976ff8f02"
server_key_version: "encryptionkey@0"
I20260929 02:11:33.806576 1578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:33.826629 1578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:33.831015 1578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:33.831311 1578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:33.849005 1578 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:40455
I20260929 02:11:33.849044 1645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:40455 every 8 connection(s)
I20260929 02:11:33.850343 1578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:33.854075 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:33.860392 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1578
I20260929 02:11:33.860585 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:33.861018 11995 external_mini_cluster.cc:1428] Setting key 0b3f4535cdd2647a019002bcce11d7b7
I20260929 02:11:33.864989 1646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a: Bootstrap starting.
I20260929 02:11:33.868659 1646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:33.869838 1646 log.cc:824] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:33.872910 1646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a: No bootstrap required, opened a new log
Sep 29 02:11:33 dist-test-slave-q06p krb5kdc[1565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:11:33.883628 1646 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1c3adcbe684de8b00baaf01368325a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40455 } }
I20260929 02:11:33.884093 1646 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:33.884197 1646 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1c3adcbe684de8b00baaf01368325a, State: Initialized, Role: FOLLOWER
I20260929 02:11:33.884848 1646 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1c3adcbe684de8b00baaf01368325a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40455 } }
I20260929 02:11:33.885141 1646 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:33.885289 1646 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:33.885411 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:33.862808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45679 (local address 127.11.182.254:40455)
0929 02:11:33.863654 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:33.863666 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:33.863707 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:11:33.865409 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:33.865430 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:33.865809 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:33.866114 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:33.867956 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:33.869298 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:33.871010 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:33.871682 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:33.871816 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:33.876545 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:33.876555 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:33.876574 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:33.876622 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:33.880789 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:33.881489 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:33.881501 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:33.881516 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:33.881734 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:33.882107 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:33.882121 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:33.882125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:33.882924 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:33.883218 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:33.883541 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 02:11:33.884021 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":377,"threads_started":1}
I20260929 02:11:33.885489 1646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:33.888558 1646 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1c3adcbe684de8b00baaf01368325a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40455 } }
I20260929 02:11:33.889070 1646 leader_election.cc:302] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf1c3adcbe684de8b00baaf01368325a; no voters:
I20260929 02:11:33.889421 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:33.889756 1646 leader_election.cc:288] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:33.889853 1651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:33.891803 1646 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:33.895164 1651 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [term 1 LEADER]: Becoming Leader. State: Replica: bf1c3adcbe684de8b00baaf01368325a, State: Running, Role: LEADER
I20260929 02:11:33.895864 1651 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1c3adcbe684de8b00baaf01368325a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40455 } }
I20260929 02:11:33.899199 1652 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf1c3adcbe684de8b00baaf01368325a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1c3adcbe684de8b00baaf01368325a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40455 } } }
I20260929 02:11:33.899665 1652 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:33.900293 1653 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf1c3adcbe684de8b00baaf01368325a. Latest consensus state: current_term: 1 leader_uuid: "bf1c3adcbe684de8b00baaf01368325a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1c3adcbe684de8b00baaf01368325a" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 40455 } } }
I20260929 02:11:33.900642 1653 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:33.901144 1665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:33.906509 1665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:33.925971 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:33.898109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56933 (local address 127.11.182.254:40455)
0929 02:11:33.898360 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:33.898369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:33.904225 (+ 5856us) server_negotiation.cc:407] Connection header received
0929 02:11:33.904467 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:33.904477 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:33.904592 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:33.904722 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:33.907418 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:33.908928 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:33.911413 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:33.912501 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:33.912646 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:33.916659 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:33.916670 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:33.916678 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:33.916729 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:33.921607 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:33.922687 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:33.922696 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:33.922702 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:33.922903 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:33.923471 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:33.923479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:33.923484 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:33.924061 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:33.924393 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:33.925150 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 02:11:33.925449 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 02:11:33.929508 1665 catalog_manager.cc:1347] Generated new cluster ID: bd769ad2d5854d76a284b1d3005c9289
I20260929 02:11:33.929697 1665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:33.942772 1665 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:33.944656 1665 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:33.964866 1665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf1c3adcbe684de8b00baaf01368325a: Generated new TSK 0
I20260929 02:11:33.966019 1665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:33.974879 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1578
I20260929 02:11:33.976274 1641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:34.069164 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1578
2026-09-29T02:11:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:34 dist-test-slave-q06p krb5kdc[1678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:34 dist-test-slave-q06p krb5kdc[1678](info): set up 2 sockets
Sep 29 02:11:34 dist-test-slave-q06p krb5kdc[1678](info): commencing operation
krb5kdc: starting...
W20260929 02:11:35.175940 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:35 dist-test-slave-q06p krb5kdc[1678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:11:35.444680 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:39821
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:35.780361 1691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:35.780654 1691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:35.780831 1691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:35.788059 1691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:35.788160 1691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:35.788323 1691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:35.788419 1691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:35.788508 1691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:35.801258 1691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:39821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:35.802588 1691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:35.803961 1691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:35.815567 1697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:35.815694 1696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:35.816149 1699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:35.816680 1691 server_base.cc:1034] running on GCE node
I20260929 02:11:35.817468 1691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:35.818994 1691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:35.820225 1691 hybrid_clock.cc:648] HybridClock initialized: now 1790647895820183 us; error 32 us; skew 500 ppm
I20260929 02:11:35.820621 1691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:35 dist-test-slave-q06p krb5kdc[1678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647895, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:35.823887 1691 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:11:35.825506 1691 webserver.cc:460] Webserver started at http://127.11.182.254:41963/ using document root <none> and password file <none>
I20260929 02:11:35.826195 1691 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:35.826325 1691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:35.826653 1691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:35.829732 1691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "44514a4390fc43a895443bbf1a055e09"
format_stamp: "Formatted at 2026-09-29 02:11:35 on dist-test-slave-q06p"
server_key: "266939f51de890c5b69c88280551f2e7"
server_key_iv: "0737119bdcc89f8d9ec5c3700c599183"
server_key_version: "encryptionkey@0"
I20260929 02:11:35.830515 1691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "44514a4390fc43a895443bbf1a055e09"
format_stamp: "Formatted at 2026-09-29 02:11:35 on dist-test-slave-q06p"
server_key: "266939f51de890c5b69c88280551f2e7"
server_key_iv: "0737119bdcc89f8d9ec5c3700c599183"
server_key_version: "encryptionkey@0"
I20260929 02:11:35.835918 1691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 02:11:35.839659 1706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:35.841154 1691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:11:35.841374 1691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "44514a4390fc43a895443bbf1a055e09"
format_stamp: "Formatted at 2026-09-29 02:11:35 on dist-test-slave-q06p"
server_key: "266939f51de890c5b69c88280551f2e7"
server_key_iv: "0737119bdcc89f8d9ec5c3700c599183"
server_key_version: "encryptionkey@0"
I20260929 02:11:35.841645 1691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:35.856057 1691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:35.860659 1691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:35.860968 1691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:35.879585 1691 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:39821
I20260929 02:11:35.879607 1758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:39821 every 8 connection(s)
I20260929 02:11:35.880970 1691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:35.884657 1759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:35.889369 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1691
I20260929 02:11:35.889554 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:35.890079 11995 external_mini_cluster.cc:1428] Setting key 0c4313df37c2baef9cb6a2022f7bd8cd
I20260929 02:11:35.896374 1759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09: Bootstrap starting.
I20260929 02:11:35.899565 1759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:35.900695 1759 log.cc:824] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:35.904208 1759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09: No bootstrap required, opened a new log
I20260929 02:11:35.904397 1762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:35.892035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53295 (local address 127.11.182.254:39821)
0929 02:11:35.892936 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:35.892948 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:35.892990 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:11:35.895095 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:35.895120 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:35.895611 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:35.895992 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:35.896557 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:35.897858 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:35.899288 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:35.900076 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:35.900277 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:35.901625 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:35.901634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:35.901650 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:35.902335 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:35.902435 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:35.902467 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:35.902585 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:35.902813 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 02:11:35.903138 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":461,"threads_started":1}
I20260929 02:11:35.908123 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:35.912570 1759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44514a4390fc43a895443bbf1a055e09" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39821 } }
I20260929 02:11:35.913084 1759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:35.913257 1759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44514a4390fc43a895443bbf1a055e09, State: Initialized, Role: FOLLOWER
I20260929 02:11:35.913968 1759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44514a4390fc43a895443bbf1a055e09" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39821 } }
I20260929 02:11:35.914263 1759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:35.914413 1759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:35.914590 1759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:35.917688 1759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44514a4390fc43a895443bbf1a055e09" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39821 } }
I20260929 02:11:35.918304 1759 leader_election.cc:302] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44514a4390fc43a895443bbf1a055e09; no voters:
I20260929 02:11:35.919040 1759 leader_election.cc:288] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:35.919138 1769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:35.919487 1762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:35.912370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55485 (local address 127.11.182.254:39821)
0929 02:11:35.912595 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:35.912601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:35.912973 (+ 372us) server_negotiation.cc:407] Connection header received
0929 02:11:35.913175 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:35.913179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:35.913264 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:35.913351 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:35.913973 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:35.914955 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:35.916752 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:35.917184 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:35.917274 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:35.918422 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:35.918427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:35.918430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:35.918446 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:35.918509 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:35.918531 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:35.918620 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:35.918813 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 02:11:35.918940 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:11:35.921130 1759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:35.923228 1769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [term 1 LEADER]: Becoming Leader. State: Replica: 44514a4390fc43a895443bbf1a055e09, State: Running, Role: LEADER
I20260929 02:11:35.923765 1769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44514a4390fc43a895443bbf1a055e09" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39821 } }
I20260929 02:11:35.926534 1771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44514a4390fc43a895443bbf1a055e09. Latest consensus state: current_term: 1 leader_uuid: "44514a4390fc43a895443bbf1a055e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44514a4390fc43a895443bbf1a055e09" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39821 } } }
I20260929 02:11:35.927016 1771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:35.927584 1770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44514a4390fc43a895443bbf1a055e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44514a4390fc43a895443bbf1a055e09" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 39821 } } }
I20260929 02:11:35.927956 1770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:35.932126 1778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:35.936218 1778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:35.944550 1778 catalog_manager.cc:1347] Generated new cluster ID: 6b767952a39c42f1af3533d9f463280c
I20260929 02:11:35.944676 1778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:35.976824 1778 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:35.978137 1778 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:35.998585 1778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44514a4390fc43a895443bbf1a055e09: Generated new TSK 0
I20260929 02:11:35.999500 1778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:36.014333 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1691
W20260929 02:11:36.015425 1754 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53295
W20260929 02:11:36.016054 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1691: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:36.016192 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1691
2026-09-29T02:11:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:36 dist-test-slave-q06p krb5kdc[1790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:36 dist-test-slave-q06p krb5kdc[1790](info): set up 2 sockets
Sep 29 02:11:36 dist-test-slave-q06p krb5kdc[1790](info): commencing operation
krb5kdc: starting...
W20260929 02:11:37.123878 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:37 dist-test-slave-q06p krb5kdc[1790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:11:37.362304 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45229
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:45229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:37.683579 1803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:37.683880 1803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:37.684047 1803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:37.691027 1803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:37.691190 1803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:37.691293 1803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:37.691401 1803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:37.691486 1803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:37.700913 1803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:45229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:37.702093 1803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:37.703354 1803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:37.715260 1808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:37.715338 1809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:37.715296 1811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:37.716024 1803 server_base.cc:1034] running on GCE node
I20260929 02:11:37.716802 1803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:37.718112 1803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:37.719347 1803 hybrid_clock.cc:648] HybridClock initialized: now 1790647897719307 us; error 55 us; skew 500 ppm
I20260929 02:11:37.719684 1803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:37 dist-test-slave-q06p krb5kdc[1790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647897, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:37.727133 1803 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:11:37.728535 1803 webserver.cc:460] Webserver started at http://127.11.182.254:44307/ using document root <none> and password file <none>
I20260929 02:11:37.729194 1803 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:37.729331 1803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:37.729641 1803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:37.732654 1803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "3f62ade447004b1a82cb6e7bf3cc4cda"
format_stamp: "Formatted at 2026-09-29 02:11:37 on dist-test-slave-q06p"
server_key: "a70857a01255cfa37fa35a3cbb3961fc"
server_key_iv: "1bb83e27ee9bcd012f50addafb895724"
server_key_version: "encryptionkey@0"
I20260929 02:11:37.733464 1803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "3f62ade447004b1a82cb6e7bf3cc4cda"
format_stamp: "Formatted at 2026-09-29 02:11:37 on dist-test-slave-q06p"
server_key: "a70857a01255cfa37fa35a3cbb3961fc"
server_key_iv: "1bb83e27ee9bcd012f50addafb895724"
server_key_version: "encryptionkey@0"
I20260929 02:11:37.738570 1803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:11:37.742018 1818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:37.743458 1803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:11:37.743650 1803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "3f62ade447004b1a82cb6e7bf3cc4cda"
format_stamp: "Formatted at 2026-09-29 02:11:37 on dist-test-slave-q06p"
server_key: "a70857a01255cfa37fa35a3cbb3961fc"
server_key_iv: "1bb83e27ee9bcd012f50addafb895724"
server_key_version: "encryptionkey@0"
I20260929 02:11:37.743854 1803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:37.749495 1803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:37.753541 1803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:37.753834 1803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:37.771742 1803 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:45229
I20260929 02:11:37.771770 1870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:45229 every 8 connection(s)
I20260929 02:11:37.773133 1803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:37.773610 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1803
I20260929 02:11:37.773788 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:37.774366 11995 external_mini_cluster.cc:1428] Setting key 8d227d8a387fe5895589701691134bd6
I20260929 02:11:37.777400 1871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:37.781823 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:37.776152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35925 (local address 127.11.182.254:45229)
0929 02:11:37.776869 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:37.776881 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:37.776919 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:11:37.778592 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:37.778613 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:37.778982 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:37.779336 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:37.780357 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:37.780362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:37.780374 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:37.780394 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:37.780454 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:37.780472 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:37.780695 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:37.780829 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:11:37.780985 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":305,"threads_started":1}
I20260929 02:11:37.785624 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:37.789050 1871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda: Bootstrap starting.
I20260929 02:11:37.792326 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:37.789858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59061 (local address 127.11.182.254:45229)
0929 02:11:37.790003 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:37.790007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:37.790446 (+ 439us) server_negotiation.cc:407] Connection header received
0929 02:11:37.790630 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:37.790637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:37.790694 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:37.790794 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:37.791717 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:37.791726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:37.791729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:11:37.791742 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:37.791777 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:11:37.791795 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:37.791907 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:37.792023 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 02:11:37.792081 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:11:37.792734 1871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:37.794004 1871 log.cc:824] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:37.796803 1871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda: No bootstrap required, opened a new log
I20260929 02:11:37.804406 1871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f62ade447004b1a82cb6e7bf3cc4cda" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45229 } }
I20260929 02:11:37.804879 1871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:11:37.804999 1871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f62ade447004b1a82cb6e7bf3cc4cda, State: Initialized, Role: FOLLOWER
I20260929 02:11:37.805533 1871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f62ade447004b1a82cb6e7bf3cc4cda" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45229 } }
I20260929 02:11:37.805775 1871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:11:37.805909 1871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:11:37.806075 1871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:37.808233 1871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f62ade447004b1a82cb6e7bf3cc4cda" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45229 } }
I20260929 02:11:37.808706 1871 leader_election.cc:302] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f62ade447004b1a82cb6e7bf3cc4cda; no voters:
I20260929 02:11:37.809273 1871 leader_election.cc:288] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:11:37.809405 1881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:37.810060 1881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [term 1 LEADER]: Becoming Leader. State: Replica: 3f62ade447004b1a82cb6e7bf3cc4cda, State: Running, Role: LEADER
I20260929 02:11:37.810636 1881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f62ade447004b1a82cb6e7bf3cc4cda" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45229 } }
I20260929 02:11:37.811466 1871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:37.813449 1883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f62ade447004b1a82cb6e7bf3cc4cda. Latest consensus state: current_term: 1 leader_uuid: "3f62ade447004b1a82cb6e7bf3cc4cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f62ade447004b1a82cb6e7bf3cc4cda" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45229 } } }
I20260929 02:11:37.813781 1883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:37.814620 1888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:37.814491 1882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f62ade447004b1a82cb6e7bf3cc4cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f62ade447004b1a82cb6e7bf3cc4cda" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45229 } } }
I20260929 02:11:37.814805 1882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:37.820248 1888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:37.828967 1888 catalog_manager.cc:1347] Generated new cluster ID: ac442ba054a84d05ac0058b093a17531
I20260929 02:11:37.829093 1888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:37.844491 1888 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:37.844887 1888 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:37.855219 1888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f62ade447004b1a82cb6e7bf3cc4cda: Generated new TSK 0
I20260929 02:11:37.855935 1888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:37.863657 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1803
W20260929 02:11:37.864813 1866 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35925
W20260929 02:11:37.865501 11995 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1803: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:11:37.865671 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1803
2026-09-29T02:11:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1849 ms)
[----------] 18 tests from AuthTokenIssuingTest (34900 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:37 dist-test-slave-q06p krb5kdc[1902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:37 dist-test-slave-q06p krb5kdc[1902](info): set up 2 sockets
Sep 29 02:11:37 dist-test-slave-q06p krb5kdc[1902](info): commencing operation
krb5kdc: starting...
W20260929 02:11:38.979970 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:39 dist-test-slave-q06p krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:11:39.231294 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45843
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:45843,127.11.182.253:42425,127.11.182.252:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:39.553854 1915 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:11:39.554260 1915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:39.554387 1915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:39.554533 1915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:39.561837 1915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:39.561965 1915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:39.562104 1915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:39.562192 1915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:39.562302 1915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:39.574617 1915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:45843,127.11.182.253:42425,127.11.182.252:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:39.576333 1915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:39.577643 1915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:39.589339 1921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:39.589383 1920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:39.589417 1923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:39.590747 1915 server_base.cc:1034] running on GCE node
I20260929 02:11:39.591562 1915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:39.592960 1915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:39.594178 1915 hybrid_clock.cc:648] HybridClock initialized: now 1790647899594123 us; error 31 us; skew 500 ppm
I20260929 02:11:39.594512 1915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:39 dist-test-slave-q06p krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:39.601389 1915 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:11:39.603267 1915 webserver.cc:460] Webserver started at http://127.11.182.254:36693/ using document root <none> and password file <none>
I20260929 02:11:39.604108 1915 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:39.604235 1915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:39.604624 1915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:39.608131 1915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "bb9c323b4485436fb5b111c5125e1f64"
format_stamp: "Formatted at 2026-09-29 02:11:39 on dist-test-slave-q06p"
server_key: "5a1b6f36edc8a739f6de0da99f63a072"
server_key_iv: "f25be4e9805e360d11f1c2ffabeee63b"
server_key_version: "encryptionkey@0"
I20260929 02:11:39.608839 1915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "bb9c323b4485436fb5b111c5125e1f64"
format_stamp: "Formatted at 2026-09-29 02:11:39 on dist-test-slave-q06p"
server_key: "5a1b6f36edc8a739f6de0da99f63a072"
server_key_iv: "f25be4e9805e360d11f1c2ffabeee63b"
server_key_version: "encryptionkey@0"
I20260929 02:11:39.613710 1915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:11:39.616950 1930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:39.618294 1915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:11:39.618580 1915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "bb9c323b4485436fb5b111c5125e1f64"
format_stamp: "Formatted at 2026-09-29 02:11:39 on dist-test-slave-q06p"
server_key: "5a1b6f36edc8a739f6de0da99f63a072"
server_key_iv: "f25be4e9805e360d11f1c2ffabeee63b"
server_key_version: "encryptionkey@0"
I20260929 02:11:39.618784 1915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:39.659746 1915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:39.663708 1915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:39.663985 1915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:39.684051 1915 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:45843
I20260929 02:11:39.684070 1982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:45843 every 8 connection(s)
I20260929 02:11:39.685544 1915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:39.689209 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:39.695945 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1915
I20260929 02:11:39.696163 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:39.695806 1983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:11:39.696904 11995 external_mini_cluster.cc:1428] Setting key 7031451cc7e28d13dcf42783b5498a58
Sep 29 02:11:39 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:11:39.719019 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.700212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.11.182.254:45843)
0929 02:11:39.700978 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:39.700988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:39.701038 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:11:39.702882 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:39.702893 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:39.703206 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:39.703434 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:39.704889 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:39.706250 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:39.707736 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:39.708318 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:39.708536 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:39.711956 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:39.711964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:39.711976 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:39.712009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:39.714687 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:39.715404 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:39.715411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:39.715413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:39.715557 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:39.715951 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:39.715962 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:39.715965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:39.716527 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:39.716730 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:39.717500 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 02:11:39.717802 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":540,"thread_start_us":293,"threads_started":1}
I20260929 02:11:39.719067 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.698969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45843 (local address 127.0.0.1:60814)
0929 02:11:39.699758 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:39.699810 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:39.702430 (+ 2620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:39.703455 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:39.703464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:39.704206 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:39.704657 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:39.704679 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:39.706411 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:39.706417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:39.707582 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:39.707595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:39.708565 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:39.708571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:39.708782 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:39.709458 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:39.709481 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:39.711730 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:39.714825 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:39.714836 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:39.714869 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:39.715255 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:39.715678 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:39.715682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:39.715685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:39.715839 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:39.716710 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:39.716722 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:39.716993 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 02:11:39.717306 (+ 313us) client_negotiation.cc:240] Negotiation successful
0929 02:11:39.717985 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":93,"thread_start_us":330,"threads_started":1}
WARNING: no policy specified for kudu/127.11.182.253@KRBTEST.COM; defaulting to no policy
I20260929 02:11:39.722667 1983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.253@KRBTEST.COM" created.
I20260929 02:11:39.723995 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.723550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41098)
0929 02:11:39.723676 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:39.723818 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 02:11:39.724293 1932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
I20260929 02:11:39.725835 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.725468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41108)
0929 02:11:39.725626 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:39.725688 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:11:39.726049 1983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:42425: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab.
Entry for principal kudu/127.11.182.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab.
I20260929 02:11:39.753959 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.253:42425
--webserver_interface=127.11.182.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab
--principal=kudu/127.11.182.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:45843,127.11.182.253:42425,127.11.182.252:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:11:39.775357 1983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } attempt: 1
I20260929 02:11:39.776618 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.776142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41120)
0929 02:11:39.776343 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:39.776479 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:11:39.777575 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.777233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41122)
0929 02:11:39.777386 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:39.777452 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:11:39.777761 1983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:42425: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
I20260929 02:11:39.846086 1983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } attempt: 2
I20260929 02:11:39.847249 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.846754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41126)
0929 02:11:39.847033 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:39.847120 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:11:39.848206 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.847853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41142)
0929 02:11:39.848008 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:39.848066 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:11:39.848382 1983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:42425: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
I20260929 02:11:39.939632 1983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } attempt: 3
I20260929 02:11:39.940701 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.940284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41158)
0929 02:11:39.940473 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:39.940568 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 02:11:39.941627 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:39.941296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41160)
0929 02:11:39.941429 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:39.941496 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:11:39.941797 1983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:42425: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
I20260929 02:11:40.085050 1983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } attempt: 4
W20260929 02:11:40.085368 1990 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:11:40.085736 1990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:40.085856 1990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:40.086027 1990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 02:11:40.086266 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.085691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41174)
0929 02:11:40.086009 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.086115 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 02:11:40.087275 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.086922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41184)
0929 02:11:40.087081 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.087162 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 02:11:40.087459 1983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:42425: Network error: Client connection negotiation failed: client connection to 127.11.182.253:42425: connect: Connection refused (error 111)
W20260929 02:11:40.093873 1990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:40.094012 1990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:40.094143 1990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:40.094233 1990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:40.094342 1990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:40.104601 1990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:45843,127.11.182.253:42425,127.11.182.252:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab
--principal=kudu/127.11.182.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.253:42425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.182.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:40.105932 1990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:40.107290 1990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:40.119488 1998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:40.119540 1996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:40.119493 1995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:40.120360 1990 server_base.cc:1034] running on GCE node
I20260929 02:11:40.121170 1990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:40.122632 1990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:40.123888 1990 hybrid_clock.cc:648] HybridClock initialized: now 1790647900123795 us; error 66 us; skew 500 ppm
I20260929 02:11:40.124236 1990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:40.127285 1990 init.cc:377] Logged in from keytab as kudu/127.11.182.253@KRBTEST.COM (short username kudu)
I20260929 02:11:40.128700 1990 webserver.cc:460] Webserver started at http://127.11.182.253:44339/ using document root <none> and password file <none>
I20260929 02:11:40.129401 1990 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:40.129532 1990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:40.129848 1990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:40.132807 1990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/data/instance:
uuid: "bd786eb93d7240b5a0d6f085c1e3af0b"
format_stamp: "Formatted at 2026-09-29 02:11:40 on dist-test-slave-q06p"
server_key: "3abbce3637e3557cc750540b1094601a"
server_key_iv: "0d3feae9e76c674e96bf968e84e3f885"
server_key_version: "encryptionkey@0"
I20260929 02:11:40.133584 1990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/wal/instance:
uuid: "bd786eb93d7240b5a0d6f085c1e3af0b"
format_stamp: "Formatted at 2026-09-29 02:11:40 on dist-test-slave-q06p"
server_key: "3abbce3637e3557cc750540b1094601a"
server_key_iv: "0d3feae9e76c674e96bf968e84e3f885"
server_key_version: "encryptionkey@0"
I20260929 02:11:40.138537 1990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:11:40.141903 2005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:40.143241 1990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:11:40.143421 1990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/wal
uuid: "bd786eb93d7240b5a0d6f085c1e3af0b"
format_stamp: "Formatted at 2026-09-29 02:11:40 on dist-test-slave-q06p"
server_key: "3abbce3637e3557cc750540b1094601a"
server_key_iv: "0d3feae9e76c674e96bf968e84e3f885"
server_key_version: "encryptionkey@0"
I20260929 02:11:40.143630 1990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:40.157624 1990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:40.161752 1990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:40.162019 1990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:40.180248 1990 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.253:42425
I20260929 02:11:40.180291 2056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.253:42425 every 8 connection(s)
I20260929 02:11:40.181564 1990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
I20260929 02:11:40.185309 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:40.189136 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 1990
I20260929 02:11:40.189343 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-1/wal/instance
I20260929 02:11:40.189801 11995 external_mini_cluster.cc:1428] Setting key 1091e41c1dc97f56ed7a7e213abe4a30
I20260929 02:11:40.192278 2057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:11:40.216537 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.195780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.11.182.254:45843)
0929 02:11:40.195973 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.195978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.197172 (+ 1194us) server_negotiation.cc:407] Connection header received
0929 02:11:40.199899 (+ 2727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.199905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.200008 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.200138 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.201647 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.202900 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.204513 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.205296 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.205476 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.209155 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.209164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.209172 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.209222 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.213226 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.213910 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.213920 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.213927 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.214098 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.214497 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.214503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.214508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.214971 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.215173 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.215793 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.216049 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.11.182.252@KRBTEST.COM; defaulting to no policy
I20260929 02:11:40.217126 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.196203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45843 (local address 127.0.0.1:60820)
0929 02:11:40.196995 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.197043 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:40.199618 (+ 2575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:40.200338 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:40.200346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:40.200915 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:40.201466 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.201482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.203046 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.203050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.204334 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.204345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.205528 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.205533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.205724 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.206424 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:40.206464 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:40.208913 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:40.213368 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.213375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.213387 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.213784 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.214227 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.214229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.214232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.214385 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.215118 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:40.215124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:40.215414 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 02:11:40.215646 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 02:11:40.215996 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":333,"threads_started":1}
I20260929 02:11:40.219210 2057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.252@KRBTEST.COM" created.
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:11:40.235764 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.220254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.11.182.253:42425)
0929 02:11:40.220876 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.220882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.220905 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:11:40.220949 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.220954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.221126 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.221279 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.222085 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.223815 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.225034 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.225966 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.226097 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.229672 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.229684 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.229698 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.229727 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.232602 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.233493 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.233500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.233502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.233624 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.234073 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.234076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.234078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.234410 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.234568 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.235067 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.235418 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":483,"thread_start_us":360,"threads_started":1}
I20260929 02:11:40.235881 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.220013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41188)
0929 02:11:40.220184 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.220204 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:40.220304 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:40.221263 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:40.221268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:40.221754 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:40.221880 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.221888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.223980 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.223985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.224872 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.224898 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.226144 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.226148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.226283 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.226958 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:40.226977 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:40.229439 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:40.232762 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.232771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.232778 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.233365 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.233737 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.233745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.233751 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.233963 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.234552 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:40.234560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:40.234741 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 02:11:40.234956 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 02:11:40.235306 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:11:40.238227 2057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:11:40.239419 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.238803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38510)
0929 02:11:40.239052 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.239204 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 02:11:40.239719 2006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
I20260929 02:11:40.241357 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.241054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38512)
0929 02:11:40.241189 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.241265 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:11:40.241677 2057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:43585: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab.
Entry for principal kudu/127.11.182.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab.
I20260929 02:11:40.253458 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.252:43585
--webserver_interface=127.11.182.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab
--principal=kudu/127.11.182.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:45843,127.11.182.253:42425,127.11.182.252:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:11:40.291003 2057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } attempt: 1
I20260929 02:11:40.292235 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.291781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38518)
0929 02:11:40.291976 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.292092 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 02:11:40.293210 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.292861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38528)
0929 02:11:40.293032 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.293112 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:11:40.293403 2057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:43585: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
I20260929 02:11:40.361605 2057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } attempt: 2
I20260929 02:11:40.362689 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.362245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38530)
0929 02:11:40.362441 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.362533 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:11:40.363701 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.363383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38536)
0929 02:11:40.363530 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.363593 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:11:40.363912 2057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:43585: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
I20260929 02:11:40.386684 1983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } attempt: 5
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:11:40.400035 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.387600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.11.182.253:42425)
0929 02:11:40.387791 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.387797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.387817 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:11:40.388031 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.388036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.388106 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.388213 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.389031 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.389743 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.391007 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.391458 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.391605 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.394822 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.394826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.394830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.394906 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.397444 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.398062 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.398068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.398072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.398208 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.398594 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.398601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.398609 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.398932 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.399080 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.399499 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.399715 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:11:40.400036 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.387485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41194)
0929 02:11:40.387710 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.387733 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:40.387872 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:40.388249 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:40.388253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:40.388686 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:40.388869 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.388878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.389882 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.389887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.390777 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.390790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.391613 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.391619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.391840 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.392548 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:40.392565 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:40.394637 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:40.397584 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.397589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.397594 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.397945 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.398328 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.398333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.398337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.398478 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.399043 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:40.399048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:40.399210 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 02:11:40.399395 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 02:11:40.399679 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:11:40.401479 1983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:11:40.402930 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.402187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38538)
0929 02:11:40.402356 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.402693 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 02:11:40.403896 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.403552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38550)
0929 02:11:40.403716 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.403792 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:11:40.404191 1983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:43585: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
I20260929 02:11:40.455104 2057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } attempt: 3
I20260929 02:11:40.455420 1983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } attempt: 1
I20260929 02:11:40.456414 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.455906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38564)
0929 02:11:40.456134 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.456254 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 02:11:40.456862 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.456004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38578)
0929 02:11:40.456295 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.456404 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260929 02:11:40.457654 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.457077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38590)
0929 02:11:40.457392 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.457492 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260929 02:11:40.457895 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.457505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38600)
0929 02:11:40.457646 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.457757 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 02:11:40.457927 2057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:43585: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
W20260929 02:11:40.458130 1983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:43585: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
I20260929 02:11:40.526340 1983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } attempt: 2
I20260929 02:11:40.527508 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.527022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38608)
0929 02:11:40.527241 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.527349 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 02:11:40.528481 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.528163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38612)
0929 02:11:40.528308 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.528389 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:11:40.528685 1983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:43585: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
W20260929 02:11:40.594269 2064 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:11:40.594691 2064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:40.594828 2064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:40.594998 2064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 02:11:40.601122 2057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } attempt: 4
I20260929 02:11:40.602270 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.601808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38626)
0929 02:11:40.602019 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.602118 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 02:11:40.602970 2064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:40.603076 2064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:40.603194 2064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:40.603286 2064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:40.603379 2064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:40.603492 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.603080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38634)
0929 02:11:40.603217 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.603329 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 02:11:40.603739 2057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:43585: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
I20260929 02:11:40.613507 2064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:45843,127.11.182.253:42425,127.11.182.252:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab
--principal=kudu/127.11.182.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.252:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.182.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:40.614773 2064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:40.616135 2064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:40.628201 2069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:40.628199 2072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:40.628224 2070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:40.629395 2064 server_base.cc:1034] running on GCE node
I20260929 02:11:40.630208 2064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:40.631863 2064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:40.633108 2064 hybrid_clock.cc:648] HybridClock initialized: now 1790647900633057 us; error 31 us; skew 500 ppm
I20260929 02:11:40.633466 2064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:11:40.634922 1983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } attempt: 3
I20260929 02:11:40.635965 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.635502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38648)
0929 02:11:40.635692 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.635781 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:40.637028 2064 init.cc:377] Logged in from keytab as kudu/127.11.182.252@KRBTEST.COM (short username kudu)
I20260929 02:11:40.637154 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.636708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38652)
0929 02:11:40.636899 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.637019 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260929 02:11:40.637439 1983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:43585: Network error: Client connection negotiation failed: client connection to 127.11.182.252:43585: connect: Connection refused (error 111)
I20260929 02:11:40.638741 2064 webserver.cc:460] Webserver started at http://127.11.182.252:42817/ using document root <none> and password file <none>
I20260929 02:11:40.639668 2064 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:40.639791 2064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:40.640133 2064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:40.643782 2064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/data/instance:
uuid: "3d6733a1636c4c938b539db5c736967f"
format_stamp: "Formatted at 2026-09-29 02:11:40 on dist-test-slave-q06p"
server_key: "054f46a3dbfffd1cf105635bd5f69aec"
server_key_iv: "4f6c52262fb1cabb39021bc2aedad301"
server_key_version: "encryptionkey@0"
I20260929 02:11:40.644590 2064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/wal/instance:
uuid: "3d6733a1636c4c938b539db5c736967f"
format_stamp: "Formatted at 2026-09-29 02:11:40 on dist-test-slave-q06p"
server_key: "054f46a3dbfffd1cf105635bd5f69aec"
server_key_iv: "4f6c52262fb1cabb39021bc2aedad301"
server_key_version: "encryptionkey@0"
I20260929 02:11:40.649982 2064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:11:40.653414 2079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:40.654911 2064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:11:40.655088 2064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/wal
uuid: "3d6733a1636c4c938b539db5c736967f"
format_stamp: "Formatted at 2026-09-29 02:11:40 on dist-test-slave-q06p"
server_key: "054f46a3dbfffd1cf105635bd5f69aec"
server_key_iv: "4f6c52262fb1cabb39021bc2aedad301"
server_key_version: "encryptionkey@0"
I20260929 02:11:40.655301 2064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:40.708048 2064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:40.712604 2064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:40.712906 2064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:40.732445 2064 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.252:43585
I20260929 02:11:40.732476 2130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.252:43585 every 8 connection(s)
I20260929 02:11:40.733847 2064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
I20260929 02:11:40.737543 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:40.738029 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 2064
I20260929 02:11:40.738232 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647540175441-11995-0/minicluster-data/master-2/wal/instance
I20260929 02:11:40.738699 11995 external_mini_cluster.cc:1428] Setting key 2f656c89f1d5d736db2f4971ffdcb0c6
I20260929 02:11:40.747535 2131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:11:40.763481 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.740527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.11.182.252:43585)
0929 02:11:40.741305 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.741320 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.741373 (+ 53us) server_negotiation.cc:407] Connection header received
0929 02:11:40.744279 (+ 2906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.744310 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.744858 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.745309 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.746631 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.748686 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.750018 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.751227 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.751553 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.755573 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.755591 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.755605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.755649 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.758933 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.759791 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.759798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.759806 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.759995 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.760333 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.760339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.760343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.761013 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.761322 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.761671 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.762105 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":311,"threads_started":1}
I20260929 02:11:40.765810 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.750397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.11.182.254:45843)
0929 02:11:40.751808 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.751813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.751828 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:11:40.751877 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.751880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.751935 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.752062 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.753165 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.753919 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.755323 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.755973 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.756115 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.759759 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.759766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.759770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.759810 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.763107 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.763863 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.763868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.763871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.764024 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.764367 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.764371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.764374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.764713 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.764869 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.765371 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.765524 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1280,"thread_start_us":1156,"threads_started":1}
I20260929 02:11:40.765991 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.750350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:45843 (local address 127.0.0.1:60824)
0929 02:11:40.751154 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.751196 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:40.751345 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:40.752127 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:40.752133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:40.752745 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:40.753006 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.753018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.754071 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.754076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.755180 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.755193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.756171 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.756175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.756382 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.757012 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:40.757032 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:40.759546 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:40.763248 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.763258 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.763262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.763709 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.764132 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.764135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.764137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.764255 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.764843 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:40.764851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:40.765057 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 02:11:40.765274 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 02:11:40.765566 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":372,"threads_started":1}
I20260929 02:11:40.767957 2131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:11:40.781165 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.768766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:42425 (local address 127.0.0.1:41200)
0929 02:11:40.768912 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.768930 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:40.769041 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:40.769445 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:40.769448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:40.769766 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:40.769894 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.769901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.770988 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.770993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.771871 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.771881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.772577 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.772583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.772761 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.773239 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:40.773254 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:40.775404 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:40.778808 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.778815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.778819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.779195 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.779586 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.779591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.779595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.779738 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.780235 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:40.780241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:40.780385 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 02:11:40.780525 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 02:11:40.780807 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 02:11:40.781213 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.768869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41200 (local address 127.11.182.253:42425)
0929 02:11:40.769179 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.769183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.769197 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:11:40.769241 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.769244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.769318 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.769414 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.770027 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.770811 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.772021 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.772432 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.772568 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.775604 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.775612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.775620 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.775663 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.778678 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.779313 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.779319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.779323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.779463 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.779846 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.779852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.779856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.780119 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.780272 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.780633 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.780841 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260929 02:11:40.781423 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.767449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.11.182.254:45843)
0929 02:11:40.767641 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.767646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.767662 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:11:40.767707 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.767711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.767786 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.767865 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.769363 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.770191 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.771480 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.771899 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.772043 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.776463 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.776468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.776471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.776506 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.778952 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.779565 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.779571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.779575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.779697 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.780052 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.780058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.780061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.780316 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.780596 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.780903 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.781121 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:11:40.782689 2131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:11:40.795463 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.783374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38672)
0929 02:11:40.783552 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.783568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:40.783664 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:40.783911 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:40.783915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:40.784260 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:40.784376 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.784384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.785312 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.785320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.786140 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.786149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.786809 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.786815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.787040 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.787584 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:40.787600 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:40.789784 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:40.792893 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.792898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.792902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.793425 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.793764 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.793771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.793776 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.793908 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.794401 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:40.794410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:40.794551 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 02:11:40.794712 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 02:11:40.795054 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":5}
I20260929 02:11:40.795490 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.783558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.11.182.252:43585)
0929 02:11:40.783677 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.783683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.783700 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:11:40.783743 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.783748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.783805 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.783897 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.784503 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.785192 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.786277 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.786663 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.786796 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.789954 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.789960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.789970 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.789996 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.792757 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.793537 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.793545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.793549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.793661 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.794006 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.794014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.794017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.794285 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.794440 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.794783 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.795192 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":184,"server-negotiator.queue_time_us":34}
I20260929 02:11:40.797102 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.783169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.11.182.253:42425)
0929 02:11:40.783343 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.783348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.783433 (+ 85us) server_negotiation.cc:407] Connection header received
0929 02:11:40.783626 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.783632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.783700 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.783798 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.784913 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.785790 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.787047 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.787483 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.787616 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.791837 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.791842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.791846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.791876 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.794353 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.795050 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.795058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.795061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.795198 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.795564 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.795570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.795573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.795944 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.796232 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.796501 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.796730 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:11:40.801254 2131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f: Bootstrap starting.
I20260929 02:11:40.801925 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:40.805225 2131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:40.806368 2131 log.cc:824] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:40.805375 1957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bb9c323b4485436fb5b111c5125e1f64"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832
I20260929 02:11:40.809110 2131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f: No bootstrap required, opened a new log
I20260929 02:11:40.814687 1983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } attempt: 4
I20260929 02:11:40.816753 2131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } }
I20260929 02:11:40.817205 2131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6733a1636c4c938b539db5c736967f, State: Initialized, Role: FOLLOWER
I20260929 02:11:40.817698 2131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } }
I20260929 02:11:40.818758 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:40.819168 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:40.819979 2131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647899, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
I20260929 02:11:40.832769 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.815364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38686)
0929 02:11:40.815663 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.815690 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:40.815816 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:40.816176 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:40.816181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:40.816669 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:40.816904 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.816917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.818428 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.818433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.819839 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.819854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.820942 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.820951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.821258 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.822145 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:40.822201 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:40.825319 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:40.829660 (+ 4341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.829665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.829671 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.830227 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.830744 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.830749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.830753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.830934 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.831717 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:40.831722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:40.831901 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 02:11:40.832039 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 02:11:40.832335 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
W20260929 02:11:40.833244 2149 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:11:40.833361 2149 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:11:40.833715 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.815457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.11.182.252:43585)
0929 02:11:40.815673 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.815683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.815744 (+ 61us) server_negotiation.cc:407] Connection header received
0929 02:11:40.815940 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.815945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.816017 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.816117 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.817051 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.818254 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.820011 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.820780 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.820899 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.825514 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.825520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.825525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.825562 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.829520 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.830392 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.830401 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.830407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.830605 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.831094 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.831098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.831101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.831540 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.833042 (+ 1502us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.833183 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.833452 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:11:40.830756 2064 master_runner.cc:186] Error getting master registration for 127.11.182.254:45843: OK, instance_id { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" instance_seqno: 1790647899678273 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:11:40.840121 2064 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:11:40.840931 1983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64: Bootstrap starting.
I20260929 02:11:40.845175 1983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:40.846554 1983 log.cc:824] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:40.850018 1983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64: No bootstrap required, opened a new log
I20260929 02:11:40.857815 1983 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } }
I20260929 02:11:40.858314 1983 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9c323b4485436fb5b111c5125e1f64, State: Initialized, Role: FOLLOWER
I20260929 02:11:40.858935 1983 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } }
I20260929 02:11:40.860075 2152 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:40.860499 2152 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:40.861420 1983 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:40.861474 1957 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 02:11:40.861720 1957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:40.864687 1957 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } }
I20260929 02:11:40.866950 2106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb9c323b4485436fb5b111c5125e1f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d6733a1636c4c938b539db5c736967f"
I20260929 02:11:40.867560 2106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:40.871208 2106 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9c323b4485436fb5b111c5125e1f64 in term 1.
I20260929 02:11:40.872021 1931 leader_election.cc:302] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d6733a1636c4c938b539db5c736967f, bb9c323b4485436fb5b111c5125e1f64; no voters:
I20260929 02:11:40.872529 2152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:40.872587 1957 leader_election.cc:288] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [CANDIDATE]: Term 1 election: Requested vote from peers bd786eb93d7240b5a0d6f085c1e3af0b (127.11.182.253:42425), 3d6733a1636c4c938b539db5c736967f (127.11.182.252:43585)
I20260929 02:11:40.885934 2152 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [term 1 LEADER]: Becoming Leader. State: Replica: bb9c323b4485436fb5b111c5125e1f64, State: Running, Role: LEADER
I20260929 02:11:40.887717 2152 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } }
I20260929 02:11:40.902909 2057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } attempt: 5
I20260929 02:11:40.906189 2160 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb9c323b4485436fb5b111c5125e1f64. Latest consensus state: current_term: 1 leader_uuid: "bb9c323b4485436fb5b111c5125e1f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:40.906605 2160 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [sys.catalog]: This master's current role is: LEADER
Sep 29 02:11:40 dist-test-slave-q06p krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647900, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
W20260929 02:11:40.916942 1936 tablet.cc:2369] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:11:40.922781 2172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:40.926146 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.903547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:43585 (local address 127.0.0.1:38698)
0929 02:11:40.903967 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:40.903987 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:40.904120 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:40.905059 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:40.905068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:40.905717 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:40.905984 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.905996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.907815 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.907822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.909623 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:40.909642 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.911219 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.911229 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:40.911536 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.912278 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:40.912307 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:40.915852 (+ 3545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:40.922256 (+ 6404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.922264 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.922270 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.922970 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.923499 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:40.923504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:40.923508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:40.923682 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:40.925011 (+ 1329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:40.925019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:40.925190 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 02:11:40.925357 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 02:11:40.925737 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260929 02:11:40.927058 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.904367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.11.182.252:43585)
0929 02:11:40.904558 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.904564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.904586 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:11:40.904661 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.904665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.904750 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.904845 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.906323 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.907619 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.909860 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.910791 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.910958 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.916133 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.916142 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.916150 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.916203 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.922049 (+ 5846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.923140 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.923149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.923156 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.923346 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.924278 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.924285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.924288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.924785 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.924933 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.925556 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.925964 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:11:40.933677 2057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b: Bootstrap starting.
I20260929 02:11:40.933723 2172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:40.937894 2057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:40.939347 2057 log.cc:824] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:40.945838 2057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b: No bootstrap required, opened a new log
I20260929 02:11:40.928882 1915 master_runner.cc:186] Error getting master registration for 127.11.182.253:42425: OK, instance_id { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" instance_seqno: 1790647900174874 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:11:40.960129 1915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:11:40.960544 2057 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } }
I20260929 02:11:40.960990 2057 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd786eb93d7240b5a0d6f085c1e3af0b, State: Initialized, Role: FOLLOWER
I20260929 02:11:40.961544 2057 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } }
W20260929 02:11:40.965366 2085 tablet.cc:2369] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:11:40.975970 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.901576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.11.182.254:45843)
0929 02:11:40.901781 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.901787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.907425 (+ 5638us) server_negotiation.cc:407] Connection header received
0929 02:11:40.907726 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.907734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.907817 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.907903 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.923011 (+ 15108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.924409 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.950558 (+ 26149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.958177 (+ 7619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.958325 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.964047 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.964056 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.964064 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.964119 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.968919 (+ 4800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.970356 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.970365 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.970371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.970564 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.971466 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.971474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.971480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.972008 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.972352 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.974764 (+ 2412us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.975352 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:11:40.983577 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.907656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.11.182.252:43585)
0929 02:11:40.927651 (+ 19995us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.927683 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.951864 (+ 24181us) server_negotiation.cc:407] Connection header received
0929 02:11:40.952009 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.952017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.952133 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.952234 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.956129 (+ 3895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.957210 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.960805 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.961515 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.961613 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.968943 (+ 7330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.968951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.968958 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.969001 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.972637 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.974828 (+ 2191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.974834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.974838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.975008 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.976375 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.976381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.976384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.976742 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.976974 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.982808 (+ 5834us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.983095 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19831,"thread_start_us":22399,"threads_started":1}
I20260929 02:11:40.988996 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:40.901013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.11.182.253:42425)
0929 02:11:40.901264 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:40.901271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:40.951275 (+ 50004us) server_negotiation.cc:407] Connection header received
0929 02:11:40.951410 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:40.951416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:40.951511 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:40.951627 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:40.954091 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.955394 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.962357 (+ 6963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:40.963165 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:40.963269 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:40.977241 (+ 13972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:40.977249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:40.977255 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:40.977299 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:40.981151 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.985788 (+ 4637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.985795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.985799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.985978 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:40.986802 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:40.986807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:40.986812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:40.987230 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:40.987448 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:40.988321 (+ 873us) server_negotiation.cc:299] Negotiation successful
0929 02:11:40.988577 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 02:11:41.007663 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:41.008210 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:41.008880 2106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [term 1 FOLLOWER]: Refusing update from remote peer bb9c323b4485436fb5b111c5125e1f64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:11:41.009392 2057 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:41.009757 2031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:41.009949 2032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb9c323b4485436fb5b111c5125e1f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b"
I20260929 02:11:41.013835 2031 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [term 1 FOLLOWER]: Refusing update from remote peer bb9c323b4485436fb5b111c5125e1f64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:11:41.014792 2160 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:11:41.015214 2152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:11:41.019798 2032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9c323b4485436fb5b111c5125e1f64 in term 1.
I20260929 02:11:41.043278 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb9c323b4485436fb5b111c5125e1f64. Latest consensus state: current_term: 1 leader_uuid: "bb9c323b4485436fb5b111c5125e1f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:41.043607 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:41.051031 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb9c323b4485436fb5b111c5125e1f64. Latest consensus state: current_term: 1 leader_uuid: "bb9c323b4485436fb5b111c5125e1f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:41.051481 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:41.053905 2181 mvcc.cc:205] Tried to move back new op lower bound from 7334493802470940672 to 7334493802056601600. Current Snapshot: MvccSnapshot[applied={T|T < 7334493802470940672}]
I20260929 02:11:41.060524 2152 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb9c323b4485436fb5b111c5125e1f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:41.060916 2152 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:41.061301 2152 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb9c323b4485436fb5b111c5125e1f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:41.061585 2152 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:41.061583 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb9c323b4485436fb5b111c5125e1f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:41.061854 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:41.068156 2172 catalog_manager.cc:1347] Generated new cluster ID: b9b83b400b23452ea0544dcf9f1352f2
I20260929 02:11:41.068297 2172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:41.068692 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb9c323b4485436fb5b111c5125e1f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9c323b4485436fb5b111c5125e1f64" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 45843 } } peers { permanent_uuid: "bd786eb93d7240b5a0d6f085c1e3af0b" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 42425 } } peers { permanent_uuid: "3d6733a1636c4c938b539db5c736967f" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 43585 } } }
I20260929 02:11:41.068965 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:41.089223 2172 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:41.091415 2172 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:41.121620 2172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb9c323b4485436fb5b111c5125e1f64: Generated new TSK 0
I20260929 02:11:41.123134 2172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:41.124079 2196 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 02:11:41.222922 2136 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.213321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.11.182.254:45843)
0929 02:11:41.213508 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.213515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.220601 (+ 7086us) server_negotiation.cc:407] Connection header received
0929 02:11:41.221030 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.221038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.221124 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.221199 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.222565 (+ 1366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60846: BlockingRecv error: recv got EOF from 127.0.0.1:60846 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 02:11:41.225111 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.213521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.11.182.253:42425)
0929 02:11:41.213715 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.213721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.223526 (+ 9805us) server_negotiation.cc:407] Connection header received
0929 02:11:41.223766 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.223773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.223861 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.223932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.224884 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41234: BlockingRecv error: recv got EOF from 127.0.0.1:41234 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 02:11:41.227788 2134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.226140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.11.182.252:43585)
0929 02:11:41.226306 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.226312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.226434 (+ 122us) server_negotiation.cc:407] Connection header received
0929 02:11:41.226562 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.226568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.226651 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.226759 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.227551 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38716: BlockingRecv error: recv got EOF from 127.0.0.1:38716 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 02:11:41.239312 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.236816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.11.182.253:42425)
0929 02:11:41.237009 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.237015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.237503 (+ 488us) server_negotiation.cc:407] Connection header received
0929 02:11:41.237744 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.237752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.237886 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.238035 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.239010 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41248: BlockingRecv error: recv got EOF from 127.0.0.1:41248 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 02:11:41.249020 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.243892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.11.182.253:42425)
0929 02:11:41.244068 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.244074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.247338 (+ 3264us) server_negotiation.cc:407] Connection header received
0929 02:11:41.247549 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.247555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.247647 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.247756 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.248778 (+ 1022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41254: BlockingRecv error: recv got EOF from 127.0.0.1:41254 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 02:11:41.259554 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.257145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.11.182.253:42425)
0929 02:11:41.257329 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.257335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.257834 (+ 499us) server_negotiation.cc:407] Connection header received
0929 02:11:41.258048 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.258054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.258201 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.258331 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.259278 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41270: BlockingRecv error: recv got EOF from 127.0.0.1:41270 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 02:11:41.274664 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.272443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.11.182.253:42425)
0929 02:11:41.272628 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.272634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.273159 (+ 525us) server_negotiation.cc:407] Connection header received
0929 02:11:41.273349 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.273355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.273459 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.273573 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.274398 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41286: BlockingRecv error: recv got EOF from 127.0.0.1:41286 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 02:11:41.289875 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.287504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.11.182.253:42425)
0929 02:11:41.287694 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.287700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.288233 (+ 533us) server_negotiation.cc:407] Connection header received
0929 02:11:41.288448 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.288454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.288592 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.288723 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.289618 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41300: BlockingRecv error: recv got EOF from 127.0.0.1:41300 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 02:11:41.315017 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.312486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.11.182.253:42425)
0929 02:11:41.312736 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.312742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.313345 (+ 603us) server_negotiation.cc:407] Connection header received
0929 02:11:41.313546 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.313553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.313662 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.313786 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.314737 (+ 951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41314: BlockingRecv error: recv got EOF from 127.0.0.1:41314 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260929 02:11:41.355352 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.352909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.11.182.253:42425)
0929 02:11:41.353133 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.353140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.353672 (+ 532us) server_negotiation.cc:407] Connection header received
0929 02:11:41.353895 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.353901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.354015 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.354136 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.355074 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41316: BlockingRecv error: recv got EOF from 127.0.0.1:41316 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 02:11:41.426780 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.424798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.11.182.253:42425)
0929 02:11:41.425002 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.425009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.425124 (+ 115us) server_negotiation.cc:407] Connection header received
0929 02:11:41.425434 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.425440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.425540 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.425657 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.426526 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41326: BlockingRecv error: recv got EOF from 127.0.0.1:41326 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 02:11:41.561474 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.559674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.11.182.253:42425)
0929 02:11:41.559843 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.559847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.560028 (+ 181us) server_negotiation.cc:407] Connection header received
0929 02:11:41.560259 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.560263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.560343 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.560435 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.561195 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41342: BlockingRecv error: recv got EOF from 127.0.0.1:41342 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 02:11:41.823539 2061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:41.821627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.11.182.253:42425)
0929 02:11:41.821848 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:41.821856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:41.822088 (+ 232us) server_negotiation.cc:407] Connection header received
0929 02:11:41.822284 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:41.822293 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:41.822389 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:41.822479 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:41.823296 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41344: BlockingRecv error: recv got EOF from 127.0.0.1:41344 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:11:41.835577 2149 catalog_manager.cc:1259] Loaded cluster ID: b9b83b400b23452ea0544dcf9f1352f2
I20260929 02:11:41.835692 2149 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f: loading cluster ID for follower catalog manager: success
I20260929 02:11:41.838295 2149 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f: acquiring CA information for follower catalog manager: success
I20260929 02:11:41.840348 2149 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d6733a1636c4c938b539db5c736967f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:11:42.128451 2196 catalog_manager.cc:1259] Loaded cluster ID: b9b83b400b23452ea0544dcf9f1352f2
I20260929 02:11:42.128563 2196 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b: loading cluster ID for follower catalog manager: success
I20260929 02:11:42.131283 2196 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b: acquiring CA information for follower catalog manager: success
I20260929 02:11:42.133163 2196 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd786eb93d7240b5a0d6f085c1e3af0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:11:42.347204 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:42.339563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.11.182.253:42425)
0929 02:11:42.340243 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:42.340250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:42.340547 (+ 297us) server_negotiation.cc:407] Connection header received
0929 02:11:42.340764 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:42.340768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:42.340947 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:42.341068 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:42.341578 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.342538 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.344622 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.345110 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.345217 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:42.345812 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:42.346377 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:42.346484 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:42.346698 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 02:11:42.346817 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":330,"threads_started":1}
I20260929 02:11:42.366961 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:42.360011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.11.182.252:43585)
0929 02:11:42.360878 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:42.360883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:42.360915 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:11:42.361042 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:42.361045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:42.361192 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:42.361342 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:42.361881 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.362624 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.364579 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.364978 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.365069 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:42.365691 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:42.366202 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:42.366311 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:42.366471 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 02:11:42.366535 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":663,"threads_started":1}
I20260929 02:11:42.392797 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:42.375385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.11.182.253:42425)
0929 02:11:42.375600 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:42.375609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:42.384668 (+ 9059us) server_negotiation.cc:407] Connection header received
0929 02:11:42.385010 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:42.385016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:42.385115 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:42.385185 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:42.386055 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.387286 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.390149 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.390910 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.391041 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:42.391763 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:42.391942 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:42.392017 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:42.392321 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 02:11:42.392411 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:11:42.398337 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:42.375097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.11.182.254:45843)
0929 02:11:42.384295 (+ 9198us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:42.384302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:42.391476 (+ 7174us) server_negotiation.cc:407] Connection header received
0929 02:11:42.391719 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:42.391723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:42.391902 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:42.392092 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:42.392739 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.393843 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.395818 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.396516 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.396671 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:42.397255 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:42.397618 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:42.397744 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:42.397870 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 02:11:42.397966 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9043,"thread_start_us":8915,"threads_started":1}
I20260929 02:11:42.399497 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:42.385003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.11.182.252:43585)
0929 02:11:42.385221 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:42.385230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:42.392651 (+ 7421us) server_negotiation.cc:407] Connection header received
0929 02:11:42.392936 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:42.392942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:42.393047 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:42.393109 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:42.393993 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.395110 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.397011 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:42.397696 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:42.397850 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:42.398573 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:42.398720 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:42.398801 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:42.399103 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:11:42.399193 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:11:42.403810 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1915
I20260929 02:11:42.404901 1978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:42.517112 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1915
W20260929 02:11:42.537997 2081 connection.cc:535] server connection from 127.0.0.1:38686 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:11:42.538429 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 1990
I20260929 02:11:42.539659 2052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:42.647708 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 1990
I20260929 02:11:42.668504 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 2064
I20260929 02:11:42.669548 2126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:42.778964 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 2064
2026-09-29T02:11:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4918 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:11:42 dist-test-slave-q06p krb5kdc[2283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:11:42 dist-test-slave-q06p krb5kdc[2283](info): set up 2 sockets
Sep 29 02:11:42 dist-test-slave-q06p krb5kdc[2283](info): commencing operation
krb5kdc: starting...
W20260929 02:11:43.928215 11995 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/test-user.keytab.
Sep 29 02:11:44 dist-test-slave-q06p krb5kdc[2283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.182.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
Entry for principal kudu/127.11.182.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab.
I20260929 02:11:44.202668 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:46453
--webserver_interface=127.11.182.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:42769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:46453,127.11.182.253:45071,127.11.182.252:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:11:44.530241 2296 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:11:44.530612 2296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:44.530737 2296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:44.530954 2296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:44.538113 2296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:44.538206 2296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:44.538353 2296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:44.538427 2296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:44.538535 2296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:44.548413 2296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:46453,127.11.182.253:45071,127.11.182.252:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.254.keytab
--principal=kudu/127.11.182.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.254:46453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.182.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:44.549671 2296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:44.551036 2296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:44.562559 2301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:44.562536 2304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:44.562604 2302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:44.563573 2296 server_base.cc:1034] running on GCE node
I20260929 02:11:44.564328 2296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:44.565668 2296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:44.566974 2296 hybrid_clock.cc:648] HybridClock initialized: now 1790647904566884 us; error 40 us; skew 500 ppm
I20260929 02:11:44.567325 2296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:44 dist-test-slave-q06p krb5kdc[2283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647904, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:44.570346 2296 init.cc:377] Logged in from keytab as kudu/127.11.182.254@KRBTEST.COM (short username kudu)
I20260929 02:11:44.571746 2296 webserver.cc:460] Webserver started at http://127.11.182.254:33925/ using document root <none> and password file <none>
I20260929 02:11:44.572371 2296 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:44.572449 2296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:44.572718 2296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:44.575634 2296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/data/instance:
uuid: "3af10c48a01144e69b89043d4e3702f0"
format_stamp: "Formatted at 2026-09-29 02:11:44 on dist-test-slave-q06p"
server_key: "847d8ee2094bb91a039ffed85aff7e58"
server_key_iv: "8411b22bb5fc070bb4cc3885cef400f5"
server_key_version: "encryptionkey@0"
I20260929 02:11:44.576557 2296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/wal/instance:
uuid: "3af10c48a01144e69b89043d4e3702f0"
format_stamp: "Formatted at 2026-09-29 02:11:44 on dist-test-slave-q06p"
server_key: "847d8ee2094bb91a039ffed85aff7e58"
server_key_iv: "8411b22bb5fc070bb4cc3885cef400f5"
server_key_version: "encryptionkey@0"
I20260929 02:11:44.584031 2296 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 02:11:44.587800 2311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:44.589109 2296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:11:44.589285 2296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/wal
uuid: "3af10c48a01144e69b89043d4e3702f0"
format_stamp: "Formatted at 2026-09-29 02:11:44 on dist-test-slave-q06p"
server_key: "847d8ee2094bb91a039ffed85aff7e58"
server_key_iv: "8411b22bb5fc070bb4cc3885cef400f5"
server_key_version: "encryptionkey@0"
I20260929 02:11:44.589502 2296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:44.610796 2296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:44.615023 2296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:44.615298 2296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:44.633806 2296 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.254:46453
I20260929 02:11:44.633826 2363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.254:46453 every 8 connection(s)
I20260929 02:11:44.635143 2296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/data/info.pb
I20260929 02:11:44.638257 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 2296
I20260929 02:11:44.638473 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-0/wal/instance
I20260929 02:11:44.638993 11995 external_mini_cluster.cc:1428] Setting key ae57a4c82361933029b5d4f270d55472
I20260929 02:11:44.639019 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:44.645511 2364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:44 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647904, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.182.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.253@KRBTEST.COM" created.
I20260929 02:11:44.669737 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.648658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.11.182.254:46453)
0929 02:11:44.650275 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:44.650286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:44.650340 (+ 54us) server_negotiation.cc:407] Connection header received
0929 02:11:44.651945 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:44.651955 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:44.652267 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:44.652472 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:44.653598 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:44.655556 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:44.657408 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:44.658445 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:44.658642 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:44.662160 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:44.662171 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:44.662186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:44.662221 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:44.665476 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:44.666147 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:44.666155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:44.666159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:44.666323 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:44.666692 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:44.666700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:44.666704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:44.667226 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:44.667429 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:44.667920 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 02:11:44.668267 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1402,"spinlock_wait_cycles":12032,"thread_start_us":1194,"threads_started":1}
I20260929 02:11:44.669744 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.648671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46453 (local address 127.0.0.1:40446)
0929 02:11:44.649632 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:44.649700 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:44.651595 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:44.652490 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:44.652494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:44.653043 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:44.653377 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:44.653393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:44.655784 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:44.655791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:44.657258 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:44.657273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:44.658674 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:44.658679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:44.658801 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:44.659532 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:44.659553 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:44.661920 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:44.665623 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:44.665634 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:44.665645 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:44.666019 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:44.666428 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:44.666436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:44.666439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:44.666592 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:44.667367 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:44.667375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:44.667571 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 02:11:44.667813 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 02:11:44.668382 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":152,"thread_start_us":389,"threads_started":1}
I20260929 02:11:44.672905 2364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:11:44.674093 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.673633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37248)
0929 02:11:44.673818 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:44.673941 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 02:11:44.674393 2312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
I20260929 02:11:44.676024 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.675590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37260)
0929 02:11:44.675763 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:44.675852 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:11:44.676234 2364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:45071: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab.
Entry for principal kudu/127.11.182.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab.
I20260929 02:11:44.697762 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.253:45071
--webserver_interface=127.11.182.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab
--principal=kudu/127.11.182.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:42769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:46453,127.11.182.253:45071,127.11.182.252:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:11:44.725602 2364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } attempt: 1
I20260929 02:11:44.726866 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.726306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37266)
0929 02:11:44.726525 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:44.726683 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 02:11:44.727911 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.727523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37268)
0929 02:11:44.727689 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:44.727772 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:11:44.728120 2364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:45071: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
I20260929 02:11:44.796363 2364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } attempt: 2
I20260929 02:11:44.797596 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.797039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37284)
0929 02:11:44.797299 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:44.797413 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 02:11:44.798734 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.798300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37290)
0929 02:11:44.798473 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:44.798559 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 02:11:44.798995 2364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:45071: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
I20260929 02:11:44.890295 2364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } attempt: 3
I20260929 02:11:44.891600 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.891091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37306)
0929 02:11:44.891311 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:44.891425 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 02:11:44.892679 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:44.892247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37316)
0929 02:11:44.892407 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:44.892516 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 02:11:44.892948 2364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:45071: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
W20260929 02:11:45.018805 2371 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:11:45.019209 2371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:45.019336 2371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:45.019474 2371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:45.026697 2371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:45.026815 2371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:45.026927 2371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:45.027017 2371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:45.027112 2371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:45.036222 2364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } attempt: 4
I20260929 02:11:45.036958 2371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:46453,127.11.182.253:45071,127.11.182.252:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.253.keytab
--principal=kudu/127.11.182.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.253:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.182.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:45.037508 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.036899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37326)
0929 02:11:45.037232 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.037342 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260929 02:11:45.038507 2371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:45.038775 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.038315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37328)
0929 02:11:45.038489 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.038618 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 02:11:45.039028 2364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.253:45071: Network error: Client connection negotiation failed: client connection to 127.11.182.253:45071: connect: Connection refused (error 111)
I20260929 02:11:45.040026 2371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:11:45.051939 2377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:45.051883 2379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:45.051946 2376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:45.052778 2371 server_base.cc:1034] running on GCE node
I20260929 02:11:45.053573 2371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:45.055003 2371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:45.056246 2371 hybrid_clock.cc:648] HybridClock initialized: now 1790647905056186 us; error 31 us; skew 500 ppm
I20260929 02:11:45.056557 2371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647905, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:45.059612 2371 init.cc:377] Logged in from keytab as kudu/127.11.182.253@KRBTEST.COM (short username kudu)
I20260929 02:11:45.060998 2371 webserver.cc:460] Webserver started at http://127.11.182.253:40799/ using document root <none> and password file <none>
I20260929 02:11:45.061717 2371 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:45.061842 2371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:45.062156 2371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:45.065027 2371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/data/instance:
uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336"
format_stamp: "Formatted at 2026-09-29 02:11:45 on dist-test-slave-q06p"
server_key: "8ebd95dc5bd2f366bbd79fbce0845547"
server_key_iv: "74d866aab4b2c75479f316045885a7fc"
server_key_version: "encryptionkey@0"
I20260929 02:11:45.065783 2371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/wal/instance:
uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336"
format_stamp: "Formatted at 2026-09-29 02:11:45 on dist-test-slave-q06p"
server_key: "8ebd95dc5bd2f366bbd79fbce0845547"
server_key_iv: "74d866aab4b2c75479f316045885a7fc"
server_key_version: "encryptionkey@0"
I20260929 02:11:45.070618 2371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 02:11:45.073988 2386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:45.075372 2371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:11:45.075551 2371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/wal
uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336"
format_stamp: "Formatted at 2026-09-29 02:11:45 on dist-test-slave-q06p"
server_key: "8ebd95dc5bd2f366bbd79fbce0845547"
server_key_iv: "74d866aab4b2c75479f316045885a7fc"
server_key_version: "encryptionkey@0"
I20260929 02:11:45.075768 2371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:45.105690 2371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:45.109542 2371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:45.109820 2371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:45.128278 2371 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.253:45071
I20260929 02:11:45.128304 2437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.253:45071 every 8 connection(s)
I20260929 02:11:45.129657 2371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/data/info.pb
I20260929 02:11:45.131443 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 2371
I20260929 02:11:45.131639 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-1/wal/instance
I20260929 02:11:45.132185 11995 external_mini_cluster.cc:1428] Setting key a497bff671f8d94c91fdb596caae7f6d
I20260929 02:11:45.134232 2438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:45.144115 2438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647905, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.182.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.182.252@KRBTEST.COM" created.
I20260929 02:11:45.165403 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.146767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.11.182.254:46453)
0929 02:11:45.146968 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.146973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.148386 (+ 1413us) server_negotiation.cc:407] Connection header received
0929 02:11:45.150235 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.150240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.150335 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.150449 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.152026 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.152712 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.154900 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.155323 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.155464 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.158970 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.158975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.158978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.159013 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.161981 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.162759 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.162767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.162774 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.162982 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.163394 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.163402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.163409 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.163824 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.163971 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.164728 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.164985 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:11:45.166110 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.147446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46453 (local address 127.0.0.1:40460)
0929 02:11:45.148223 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.148260 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:45.149953 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:45.150690 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:45.150699 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:45.151295 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:45.151845 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.151863 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.152881 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.152885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.154728 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.154741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.155560 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.155567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.155809 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.156501 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:45.156524 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:45.158690 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:45.162146 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.162155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.162170 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.162626 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.163111 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.163114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.163117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.163283 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.163948 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:45.163953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:45.164313 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 02:11:45.164574 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 02:11:45.164984 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":324,"threads_started":1}
I20260929 02:11:45.167964 2438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647905, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:11:45.184953 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.168837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:37332)
0929 02:11:45.169007 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.169028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:45.169139 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:45.170332 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:45.170336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:45.170702 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:45.170838 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.170901 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.173036 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.173039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.173870 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.173880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.175120 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.175125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.175312 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.175901 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:45.175917 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:45.178140 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:45.182502 (+ 4362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.182506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.182509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.182912 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.183326 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.183329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.183331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.183461 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.184201 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:45.184207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:45.184362 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 02:11:45.184489 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 02:11:45.184689 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":47}
I20260929 02:11:45.185473 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.169154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.11.182.253:45071)
0929 02:11:45.169842 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.169847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.169873 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:11:45.169914 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.169919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.170156 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.170322 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.171068 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.172896 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.174007 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.174967 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.175097 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.178361 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.178376 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.178396 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.178437 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.182366 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.183054 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.183065 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.183072 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.183219 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.183567 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.183574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.183580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.184054 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.184285 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.184694 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.184981 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":542,"thread_start_us":391,"threads_started":1}
I20260929 02:11:45.188091 2438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:11:45.189165 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.188733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:42434)
0929 02:11:45.188922 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.189015 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 02:11:45.189390 2388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
I20260929 02:11:45.190781 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.190414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:42436)
0929 02:11:45.190607 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.190684 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 02:11:45.191076 2438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:38295: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.182.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab.
Entry for principal kudu/127.11.182.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab.
I20260929 02:11:45.197607 11995 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkd1a/build/asan/bin/kudu
/tmp/dist-test-taskizkd1a/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.252:38295
--webserver_interface=127.11.182.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab
--principal=kudu/127.11.182.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.182.212:42769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.182.254:46453,127.11.182.253:45071,127.11.182.252:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:11:45.240450 2438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } attempt: 1
I20260929 02:11:45.241770 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.241170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:42446)
0929 02:11:45.241419 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.241551 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 02:11:45.242812 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.242407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:42462)
0929 02:11:45.242553 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.242657 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:11:45.243081 2438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:38295: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
I20260929 02:11:45.311343 2438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } attempt: 2
I20260929 02:11:45.312544 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.312023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33516)
0929 02:11:45.312241 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.312344 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 02:11:45.313537 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.313178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33522)
0929 02:11:45.313321 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.313399 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:11:45.313745 2438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:38295: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
I20260929 02:11:45.338333 2364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } attempt: 5
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647904, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:11:45.353178 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.339186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:38790)
0929 02:11:45.339411 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.339435 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:45.339578 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:45.339935 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:45.339940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:45.340450 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:45.340699 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.340712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.341662 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.341668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.343022 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.343037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.343756 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.343763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.343995 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.344621 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:45.344643 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:45.347273 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:45.350948 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.350952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.350955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.351293 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.351651 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.351654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.351656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.351781 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.352441 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:45.352445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:45.352621 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 02:11:45.352733 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:11:45.352933 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:11:45.353574 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.339416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.11.182.253:45071)
0929 02:11:45.339588 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.339592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.339611 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:11:45.339707 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.339711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.339772 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.339866 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.340842 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.341533 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.343167 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.343583 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.343679 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.347472 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.347478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.347483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.347521 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.350774 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.351404 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.351409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.351413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.351549 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.351888 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.351895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.351909 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.352322 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.352493 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.352913 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.353189 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:11:45.354624 2364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:11:45.355621 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.355266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33528)
0929 02:11:45.355421 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.355499 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 02:11:45.356515 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.356232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33536)
0929 02:11:45.356369 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.356428 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:11:45.356730 2364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:38295: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
I20260929 02:11:45.405020 2438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } attempt: 3
I20260929 02:11:45.406515 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.405811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33548)
0929 02:11:45.406091 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.406231 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 02:11:45.407828 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.407243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33558)
0929 02:11:45.407505 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.407618 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 02:11:45.408004 2364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } attempt: 1
W20260929 02:11:45.408169 2438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:38295: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
I20260929 02:11:45.409121 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.408723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33560)
0929 02:11:45.408906 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.408983 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:11:45.409997 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.409687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33568)
0929 02:11:45.409826 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.409899 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:11:45.410223 2364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:38295: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
I20260929 02:11:45.478446 2364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } attempt: 2
I20260929 02:11:45.479954 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.479367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33578)
0929 02:11:45.479575 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.479721 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 02:11:45.480975 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.480632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33592)
0929 02:11:45.480787 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.480870 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:11:45.481197 2364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:38295: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
W20260929 02:11:45.529601 2445 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:11:45.529983 2445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:11:45.530112 2445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:11:45.530274 2445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:11:45.537330 2445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:11:45.537477 2445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:11:45.537577 2445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:11:45.537672 2445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:11:45.537768 2445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:11:45.547228 2445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.182.212:42769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.182.254:46453,127.11.182.253:45071,127.11.182.252:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/krb5kdc/kudu_127.11.182.252.keytab
--principal=kudu/127.11.182.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.182.252:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.182.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:11:45.548513 2445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:11:45.549826 2445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 02:11:45.551396 2438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } attempt: 4
I20260929 02:11:45.552563 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.552107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33600)
0929 02:11:45.552314 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.552405 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:11:45.553479 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.553191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33612)
0929 02:11:45.553326 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.553393 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:11:45.553670 2438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:38295: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
W20260929 02:11:45.561805 2454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:45.561878 2452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:11:45.562098 2450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:11:45.562951 2445 server_base.cc:1034] running on GCE node
I20260929 02:11:45.563762 2445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:11:45.565191 2445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:11:45.566433 2445 hybrid_clock.cc:648] HybridClock initialized: now 1790647905566351 us; error 25 us; skew 500 ppm
I20260929 02:11:45.566769 2445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647905, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:11:45.569728 2445 init.cc:377] Logged in from keytab as kudu/127.11.182.252@KRBTEST.COM (short username kudu)
I20260929 02:11:45.571062 2445 webserver.cc:460] Webserver started at http://127.11.182.252:37171/ using document root <none> and password file <none>
I20260929 02:11:45.571779 2445 fs_manager.cc:359] Metadata directory not provided
I20260929 02:11:45.571861 2445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:11:45.572103 2445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:11:45.574972 2445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/data/instance:
uuid: "a90ce44a9af2497e9286e9cff3714026"
format_stamp: "Formatted at 2026-09-29 02:11:45 on dist-test-slave-q06p"
server_key: "49872b3268cb771872a1dac97f068273"
server_key_iv: "1f2be408f7c015eac7eff570d51d583c"
server_key_version: "encryptionkey@0"
I20260929 02:11:45.575719 2445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/wal/instance:
uuid: "a90ce44a9af2497e9286e9cff3714026"
format_stamp: "Formatted at 2026-09-29 02:11:45 on dist-test-slave-q06p"
server_key: "49872b3268cb771872a1dac97f068273"
server_key_iv: "1f2be408f7c015eac7eff570d51d583c"
server_key_version: "encryptionkey@0"
I20260929 02:11:45.580716 2445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:11:45.584015 2461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:11:45.585283 2445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:11:45.585420 2445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/wal
uuid: "a90ce44a9af2497e9286e9cff3714026"
format_stamp: "Formatted at 2026-09-29 02:11:45 on dist-test-slave-q06p"
server_key: "49872b3268cb771872a1dac97f068273"
server_key_iv: "1f2be408f7c015eac7eff570d51d583c"
server_key_version: "encryptionkey@0"
I20260929 02:11:45.585598 2445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:11:45.587409 2364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } attempt: 3
I20260929 02:11:45.588505 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.587953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33628)
0929 02:11:45.588193 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.588293 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 02:11:45.589586 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.589160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33640)
0929 02:11:45.589352 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.589457 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 02:11:45.589846 2364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.182.252:38295: Network error: Client connection negotiation failed: client connection to 127.11.182.252:38295: connect: Connection refused (error 111)
I20260929 02:11:45.606992 2445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:11:45.611341 2445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:11:45.611603 2445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:11:45.629966 2445 rpc_server.cc:307] RPC server started. Bound to: 127.11.182.252:38295
I20260929 02:11:45.629990 2512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.182.252:38295 every 8 connection(s)
I20260929 02:11:45.631209 2445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/data/info.pb
I20260929 02:11:45.632582 11995 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkd1a/build/asan/bin/kudu as pid 2445
I20260929 02:11:45.632768 11995 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkd1a/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647540175441-11995-0/minicluster-data/master-2/wal/instance
I20260929 02:11:45.633288 11995 external_mini_cluster.cc:1428] Setting key 63ad011842e15d32588bf0e3552ca859
I20260929 02:11:45.635586 2513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:11:45.644114 2513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647905, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
I20260929 02:11:45.657742 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.635232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.11.182.252:38295)
0929 02:11:45.636015 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.636029 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.636073 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:11:45.638287 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.638314 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.638821 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.639227 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.641011 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.643048 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.644827 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.645547 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.645665 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.650543 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.650551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.650573 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.650606 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.653826 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.654519 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.654527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.654535 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.654697 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.655157 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.655161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.655163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.655656 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.655952 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.656391 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.656796 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":402,"threads_started":1}
I20260929 02:11:45.663411 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.646828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.11.182.254:46453)
0929 02:11:45.647065 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.647072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.647861 (+ 789us) server_negotiation.cc:407] Connection header received
0929 02:11:45.648062 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.648066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.648127 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.648204 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.649640 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.650636 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.652202 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.652626 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.652764 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.655913 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.655918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.655921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.655953 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.659151 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.661147 (+ 1996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.661155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.661161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.661333 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.661857 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.661862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.661866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.662144 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.662337 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.662820 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.663058 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 02:11:45.663437 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.646943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.254:46453 (local address 127.0.0.1:52344)
0929 02:11:45.647708 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.647752 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:45.647935 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:45.648304 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:45.648311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:45.649177 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:45.649422 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.649430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.650821 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.650826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.652052 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.652065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.652826 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.652834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.653027 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.653597 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:45.653614 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:45.655722 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:45.659291 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.659300 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.659305 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.659910 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.661479 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.661484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.661488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.661673 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.662298 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:45.662305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:45.662496 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 02:11:45.662680 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 02:11:45.663024 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":347,"threads_started":1}
I20260929 02:11:45.664683 2513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.254@KRBTEST.COM
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647905, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:11:45.677191 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.661912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.11.182.254:46453)
0929 02:11:45.662515 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.662519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.662537 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:11:45.662574 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.662577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.662628 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.662752 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.664115 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.665135 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.666666 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.667137 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.667279 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.671314 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.671322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.671332 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.671360 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.674185 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.674951 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.674989 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.674995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.675249 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.675738 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.675742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.675744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.676043 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.676318 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.676544 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.676779 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":333,"threads_started":1}
I20260929 02:11:45.678238 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.665326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.253:45071 (local address 127.0.0.1:38806)
0929 02:11:45.665611 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.665627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:45.665710 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:45.666039 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:45.666043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:45.666453 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:45.666586 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.666593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.667666 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.667672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.668704 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.668715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.669427 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.669432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.669603 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.670085 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:45.670113 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:45.672162 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:45.675512 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.675516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.675520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.675958 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.676368 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.676372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.676376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.676542 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.677310 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:45.677314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:45.677474 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 02:11:45.677617 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 02:11:45.677879 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 02:11:45.678295 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.665425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.11.182.253:45071)
0929 02:11:45.665588 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.665593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.665701 (+ 108us) server_negotiation.cc:407] Connection header received
0929 02:11:45.665796 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.665802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.665881 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.665984 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.666729 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.667482 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.668859 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.669266 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.669422 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.672353 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.672357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.672360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.672393 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.675364 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.676072 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.676078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.676083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.676250 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.676655 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.676659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.676662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.677163 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.677335 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.677716 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.677926 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:11:45.679653 2513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647905, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.252@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.182.253@KRBTEST.COM
I20260929 02:11:45.694069 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.680334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33658)
0929 02:11:45.680501 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.680519 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:45.680614 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:45.680909 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:45.680913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:45.681353 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:45.681474 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.681481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.682667 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.682674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.683915 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.683926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.684907 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.684914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.685108 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.685642 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:45.685659 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:45.687774 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:45.691426 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.691435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.691439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.691812 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.692215 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.692222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.692226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.692367 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.692957 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:45.692962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:45.693131 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 02:11:45.693322 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 02:11:45.693591 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 02:11:45.694156 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.680471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.11.182.252:38295)
0929 02:11:45.680610 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.680614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.680628 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:11:45.680722 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.680726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.680785 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.680854 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.681605 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.682542 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.684057 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.684717 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.684868 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.687963 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.687971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.687976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.688017 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.691277 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.691941 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.691950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.691955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.692092 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.692507 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.692515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.692518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.692807 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.692991 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.693429 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.693761 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":37}
I20260929 02:11:45.694551 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.678988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.11.182.253:45071)
0929 02:11:45.679136 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.679139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.679253 (+ 114us) server_negotiation.cc:407] Connection header received
0929 02:11:45.679470 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.679473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.679524 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.679626 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.681182 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.682098 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.683709 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.684278 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.684430 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.688958 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.688963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.688966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.688996 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.691682 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.692352 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.692357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.692360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.692527 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.692920 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.692924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.692928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.693233 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.693510 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.693981 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.694209 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:11:45.699234 11995 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:11:45.699801 2513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026: Bootstrap starting.
I20260929 02:11:45.702970 2513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:45.703966 2513 log.cc:824] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:45.701531 2339 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3af10c48a01144e69b89043d4e3702f0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52356
I20260929 02:11:45.706526 2513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026: No bootstrap required, opened a new log
I20260929 02:11:45.713366 2513 raft_consensus.cc:348] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } }
I20260929 02:11:45.713703 2513 raft_consensus.cc:729] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90ce44a9af2497e9286e9cff3714026, State: Initialized, Role: FOLLOWER
I20260929 02:11:45.714110 2513 consensus_queue.cc:260] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } }
I20260929 02:11:45.714989 2520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:45.715302 2520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:45.716019 2513 sys_catalog.cc:564] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:11:45.725373 2531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:11:45.725507 2531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:11:45.723222 2445 master_runner.cc:186] Error getting master registration for 127.11.182.254:46453: OK, instance_id { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" instance_seqno: 1790647904628485 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:11:45.731158 2445 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:11:45.767081 2364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } attempt: 4
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647904, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.254@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
I20260929 02:11:45.783504 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.768108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.11.182.252:38295)
0929 02:11:45.768336 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.768342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.768369 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:11:45.768446 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.768453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.768534 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.768649 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.769731 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.770890 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.772737 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.773501 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.773633 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.777187 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.777192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.777195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.777228 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.780603 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.781379 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.781384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.781388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.781519 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.781955 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.781959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.781963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.782329 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.782497 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.782932 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.783150 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:11:45.783516 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.767858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33668)
0929 02:11:45.768094 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.768115 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:45.768254 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:45.768711 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:45.768719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:45.769289 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:45.769554 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.769565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.771035 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.771041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.772581 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.772595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.773652 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.773659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.773842 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.774509 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:45.774527 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:45.776993 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:45.780739 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.780745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.780749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.781244 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.781646 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.781652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.781655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.781835 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.782458 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:45.782464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:45.782634 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 02:11:45.782808 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 02:11:45.783121 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:11:45.788064 2364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0: Bootstrap starting.
I20260929 02:11:45.792114 2364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:45.793394 2364 log.cc:824] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:45.796360 2364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0: No bootstrap required, opened a new log
I20260929 02:11:45.801368 2364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } }
I20260929 02:11:45.801728 2364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af10c48a01144e69b89043d4e3702f0, State: Initialized, Role: FOLLOWER
I20260929 02:11:45.802237 2364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } }
I20260929 02:11:45.803191 2534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:45.803583 2534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:45.804330 2364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:45.804448 2339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 02:11:45.804648 2339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:45.808099 2339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } }
I20260929 02:11:45.810118 2339 leader_election.cc:288] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [CANDIDATE]: Term 1 election: Requested vote from peers 2f29da5de7d648bfb2dd2bf4f5ee6336 (127.11.182.253:45071), a90ce44a9af2497e9286e9cff3714026 (127.11.182.252:38295)
I20260929 02:11:45.810798 2488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3af10c48a01144e69b89043d4e3702f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a90ce44a9af2497e9286e9cff3714026"
I20260929 02:11:45.811437 2488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:45.815757 2488 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af10c48a01144e69b89043d4e3702f0 in term 1.
I20260929 02:11:45.816408 2313 leader_election.cc:302] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af10c48a01144e69b89043d4e3702f0, a90ce44a9af2497e9286e9cff3714026; no voters:
I20260929 02:11:45.816897 2534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:11:45.817706 2534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [term 1 LEADER]: Becoming Leader. State: Replica: 3af10c48a01144e69b89043d4e3702f0, State: Running, Role: LEADER
I20260929 02:11:45.818290 2534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } }
I20260929 02:11:45.821898 2542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af10c48a01144e69b89043d4e3702f0. Latest consensus state: current_term: 1 leader_uuid: "3af10c48a01144e69b89043d4e3702f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:45.822456 2542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:45.835584 2544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:11:45.849912 2544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:11:45.852855 2438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } attempt: 5
W20260929 02:11:45.863623 2467 tablet.cc:2369] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:11:45 dist-test-slave-q06p krb5kdc[2283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647905, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.182.253@KRBTEST.COM for kudu/127.11.182.252@KRBTEST.COM
I20260929 02:11:45.867522 2488 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [term 1 FOLLOWER]: Refusing update from remote peer 3af10c48a01144e69b89043d4e3702f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 02:11:45.868242 2317 tablet.cc:2369] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:11:45.869415 2542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:11:45.876437 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.853409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.182.252:38295 (local address 127.0.0.1:33670)
0929 02:11:45.853770 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 02:11:45.853793 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:11:45.853899 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:11:45.855529 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:11:45.855538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:11:45.856222 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:11:45.856472 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.856487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.858271 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.858280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.860094 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:11:45.860114 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.861588 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.861597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:11:45.861955 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.862776 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:11:45.862805 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:11:45.866463 (+ 3658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:11:45.871921 (+ 5458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.871926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.871930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.872592 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.873716 (+ 1124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:11:45.873724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:11:45.873731 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:11:45.873924 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:11:45.875173 (+ 1249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:11:45.875180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:11:45.875382 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 02:11:45.875574 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 02:11:45.875926 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260929 02:11:45.878490 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.854176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.11.182.252:38295)
0929 02:11:45.854418 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.854427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.854451 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:11:45.854525 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.854532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.854626 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.854741 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.856692 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.858043 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.860296 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.861195 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.861349 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.866929 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.866938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.866946 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.866998 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.871753 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.873237 (+ 1484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.873247 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.873253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.873468 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.874465 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.874473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.874479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.874910 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.875101 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.877685 (+ 2584us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.877981 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:11:45.889791 2438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336: Bootstrap starting.
I20260929 02:11:45.891706 2520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af10c48a01144e69b89043d4e3702f0. Latest consensus state: current_term: 1 leader_uuid: "3af10c48a01144e69b89043d4e3702f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:45.892047 2520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:45.896138 2556 mvcc.cc:205] Tried to move back new op lower bound from 7334493822420406272 to 7334493822236954624. Current Snapshot: MvccSnapshot[applied={T|T < 7334493822420406272}]
I20260929 02:11:45.896824 2438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336: Neither blocks nor log segments found. Creating new log.
I20260929 02:11:45.898131 2542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3af10c48a01144e69b89043d4e3702f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:45.898430 2542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:45.898659 2557 mvcc.cc:205] Tried to move back new op lower bound from 7334493822420406272 to 7334493822236954624. Current Snapshot: MvccSnapshot[applied={T|T < 7334493822420406272}]
I20260929 02:11:45.900378 2438 log.cc:824] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336: Log is configured to *not* fsync() on all Append() calls
I20260929 02:11:45.901232 2520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3af10c48a01144e69b89043d4e3702f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:45.901496 2520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:45.911799 2544 catalog_manager.cc:1347] Generated new cluster ID: ce5d7cf6958146fcaeed942b1b1afa4b
I20260929 02:11:45.911931 2544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:11:45.922339 2438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336: No bootstrap required, opened a new log
I20260929 02:11:45.924935 2296 master_runner.cc:186] Error getting master registration for 127.11.182.253:45071: OK, instance_id { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" instance_seqno: 1790647905123143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:11:45.941854 2296 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:11:45.947302 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.849532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52368 (local address 127.11.182.254:46453)
0929 02:11:45.849743 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.849750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.855016 (+ 5266us) server_negotiation.cc:407] Connection header received
0929 02:11:45.855105 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.855110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.855204 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.855315 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.901986 (+ 46671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.903440 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.916172 (+ 12732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.917102 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.917472 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.936185 (+ 18713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.936192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.936198 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.936244 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.940653 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.942291 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.942296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.942299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.942451 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.942992 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.942998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.943002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.943382 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.943662 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.946641 (+ 2979us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.946923 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:11:45.964071 2544 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:11:45.965902 2544 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:11:45.966825 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.849246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.11.182.253:45071)
0929 02:11:45.849439 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.849445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.920542 (+ 71097us) server_negotiation.cc:407] Connection header received
0929 02:11:45.925262 (+ 4720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.925269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.925351 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.925421 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.932304 (+ 6883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.933459 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.944964 (+ 11505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.945629 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.945752 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.957245 (+ 11493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.957253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.957257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.957306 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.960911 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.961888 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.961896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.961901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.962078 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.964392 (+ 2314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.964399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.964403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.964841 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.965094 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.965606 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.966134 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:11:45.972685 2438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } }
I20260929 02:11:45.973106 2438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f29da5de7d648bfb2dd2bf4f5ee6336, State: Initialized, Role: FOLLOWER
I20260929 02:11:45.973635 2438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } }
I20260929 02:11:45.977255 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:45.919511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.11.182.252:38295)
0929 02:11:45.919743 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:45.919751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:45.940873 (+ 21122us) server_negotiation.cc:407] Connection header received
0929 02:11:45.942307 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:45.942314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:45.942429 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:45.942510 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:45.944571 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.945662 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.959015 (+ 13353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:45.959821 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:45.959940 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:45.966010 (+ 6070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:11:45.966018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:11:45.966022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:11:45.966075 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:11:45.969937 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.971023 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.971034 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.971038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.971188 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:11:45.974274 (+ 3086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:11:45.974279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:11:45.974282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:11:45.974799 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:11:45.975126 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:45.976426 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0929 02:11:45.976730 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:11:45.981819 2563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:45.982213 2563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:45.983970 2438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:11:45.984395 2412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:11:45.989070 2412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [term 1 FOLLOWER]: Refusing update from remote peer 3af10c48a01144e69b89043d4e3702f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:11:45.990183 2542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:11:45.990300 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3af10c48a01144e69b89043d4e3702f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336"
I20260929 02:11:45.997882 2413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af10c48a01144e69b89043d4e3702f0 in term 1.
I20260929 02:11:46.000118 2544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0: Generated new TSK 0
I20260929 02:11:46.001140 2544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:11:46.062213 2563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af10c48a01144e69b89043d4e3702f0. Latest consensus state: current_term: 1 leader_uuid: "3af10c48a01144e69b89043d4e3702f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:46.062588 2563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:46.063140 2572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3af10c48a01144e69b89043d4e3702f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:46.063552 2572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:11:46.065330 2534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3af10c48a01144e69b89043d4e3702f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af10c48a01144e69b89043d4e3702f0" member_type: VOTER last_known_addr { host: "127.11.182.254" port: 46453 } } peers { permanent_uuid: "2f29da5de7d648bfb2dd2bf4f5ee6336" member_type: VOTER last_known_addr { host: "127.11.182.253" port: 45071 } } peers { permanent_uuid: "a90ce44a9af2497e9286e9cff3714026" member_type: VOTER last_known_addr { host: "127.11.182.252" port: 38295 } } }
I20260929 02:11:46.065696 2534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af10c48a01144e69b89043d4e3702f0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:11:46.096422 2580 catalog_manager.cc:1259] Loaded cluster ID: ce5d7cf6958146fcaeed942b1b1afa4b
I20260929 02:11:46.096609 2580 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336: loading cluster ID for follower catalog manager: success
I20260929 02:11:46.100514 2580 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336: acquiring CA information for follower catalog manager: success
I20260929 02:11:46.111923 2580 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f29da5de7d648bfb2dd2bf4f5ee6336: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:11:46.126106 11995 mini_kdc.cc:348] Time spent kdestroy: real 0.108s user 0.000s sys 0.025s
W20260929 02:11:46.188139 2518 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.162677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.11.182.254:46453)
0929 02:11:46.162943 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.162950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.186047 (+ 23097us) server_negotiation.cc:407] Connection header received
0929 02:11:46.186369 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.186376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.186471 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.186551 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.187818 (+ 1267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52378: BlockingRecv error: recv got EOF from 127.0.0.1:52378 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260929 02:11:46.190868 2442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.169833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.11.182.253:45071)
0929 02:11:46.170063 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.170070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.188876 (+ 18806us) server_negotiation.cc:407] Connection header received
0929 02:11:46.189342 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.189353 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.189462 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.189538 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.190603 (+ 1065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38830: BlockingRecv error: recv got EOF from 127.0.0.1:38830 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 02:11:46.194540 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.192007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.11.182.252:38295)
0929 02:11:46.192227 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.192233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.192846 (+ 613us) server_negotiation.cc:407] Connection header received
0929 02:11:46.193088 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.193094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.193170 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.193242 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.194248 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33678: BlockingRecv error: recv got EOF from 127.0.0.1:33678 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:11:46.249601 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.232802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.11.182.253:45071)
0929 02:11:46.233051 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.233060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.240898 (+ 7838us) server_negotiation.cc:407] Connection header received
0929 02:11:46.241473 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.241485 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.241691 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.241830 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.242550 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.243721 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.245845 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.246569 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.246681 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.247479 (+ 798us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.248266 (+ 787us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.248383 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.249108 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.249218 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 02:11:46.301537 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.287329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.11.182.252:38295)
0929 02:11:46.287561 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.287568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.299491 (+ 11923us) server_negotiation.cc:407] Connection header received
0929 02:11:46.299777 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.299786 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.299952 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.300039 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.301280 (+ 1241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33682: BlockingRecv error: recv got EOF from 127.0.0.1:33682 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:11:46.354952 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.337038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38846 (local address 127.11.182.253:45071)
0929 02:11:46.337230 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.337236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.347829 (+ 10593us) server_negotiation.cc:407] Connection header received
0929 02:11:46.348378 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.348385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.348485 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.348557 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.349226 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.350394 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.352496 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.353235 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.353337 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.354032 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.354212 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.354280 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.354514 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.354620 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 02:11:46.365018 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.362803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.11.182.252:38295)
0929 02:11:46.363022 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.363028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.363559 (+ 531us) server_negotiation.cc:407] Connection header received
0929 02:11:46.363760 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.363766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.363862 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.363971 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.364787 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33686: BlockingRecv error: recv got EOF from 127.0.0.1:33686 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 02:11:46.378034 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.370905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.11.182.253:45071)
0929 02:11:46.371102 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.371108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.371604 (+ 496us) server_negotiation.cc:407] Connection header received
0929 02:11:46.371841 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.371848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.371964 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.372066 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.372559 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.373747 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.375794 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.376498 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.376639 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.377158 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.377328 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.377427 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.377606 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.377694 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 02:11:46.387596 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.385474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.11.182.252:38295)
0929 02:11:46.385660 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.385666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.386065 (+ 399us) server_negotiation.cc:407] Connection header received
0929 02:11:46.386277 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.386284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.386374 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.386482 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.387340 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33700: BlockingRecv error: recv got EOF from 127.0.0.1:33700 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:11:46.404081 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.396771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.11.182.253:45071)
0929 02:11:46.397013 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.397021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.397541 (+ 520us) server_negotiation.cc:407] Connection header received
0929 02:11:46.397748 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.397754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.397852 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.397952 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.398460 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.399708 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.401957 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.402647 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.402775 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.403296 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.403438 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.403527 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.403711 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.403793 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260929 02:11:46.425093 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.415680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.11.182.252:38295)
0929 02:11:46.415943 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.415949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.423487 (+ 7538us) server_negotiation.cc:407] Connection header received
0929 02:11:46.423697 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.423703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.423813 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.423937 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.424848 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33716: BlockingRecv error: recv got EOF from 127.0.0.1:33716 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
I20260929 02:11:46.443706 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.436501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.11.182.253:45071)
0929 02:11:46.436720 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.436726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.437299 (+ 573us) server_negotiation.cc:407] Connection header received
0929 02:11:46.437517 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.437527 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.437635 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.437755 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.438242 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.439504 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.441515 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.442177 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.442334 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.442789 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.442987 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.443091 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.443267 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.443353 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 02:11:46.454782 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.452534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.11.182.252:38295)
0929 02:11:46.452740 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.452750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.453279 (+ 529us) server_negotiation.cc:407] Connection header received
0929 02:11:46.453469 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.453475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.453576 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.453686 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.454527 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33720: BlockingRecv error: recv got EOF from 127.0.0.1:33720 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:11:46.483150 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.475795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.11.182.253:45071)
0929 02:11:46.476011 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.476018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.476620 (+ 602us) server_negotiation.cc:407] Connection header received
0929 02:11:46.476819 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.476826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.476925 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.477035 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.477544 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.478872 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.480897 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.481600 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.481720 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.482272 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.482411 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.482499 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.482720 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.482816 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 02:11:46.495052 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.492628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.11.182.252:38295)
0929 02:11:46.492846 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.492852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.493443 (+ 591us) server_negotiation.cc:407] Connection header received
0929 02:11:46.493678 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.493684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.493795 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.493914 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.494786 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33726: BlockingRecv error: recv got EOF from 127.0.0.1:33726 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:11:46.537317 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.531022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.11.182.253:45071)
0929 02:11:46.531279 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.531285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.531596 (+ 311us) server_negotiation.cc:407] Connection header received
0929 02:11:46.531812 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.531816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.531903 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.531987 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.532494 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.533263 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.535324 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.535830 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.535945 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.536517 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.536616 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.536693 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.536874 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.536944 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 02:11:46.545586 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.543456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.11.182.252:38295)
0929 02:11:46.543677 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.543683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.544009 (+ 326us) server_negotiation.cc:407] Connection header received
0929 02:11:46.544164 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.544168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.544243 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.544333 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.545305 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33740: BlockingRecv error: recv got EOF from 127.0.0.1:33740 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 02:11:46.620060 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.613688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.11.182.253:45071)
0929 02:11:46.614180 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.614189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.614386 (+ 197us) server_negotiation.cc:407] Connection header received
0929 02:11:46.614571 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.614577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.614663 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.614768 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.615287 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.616490 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.618068 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.618768 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.618920 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.619301 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.619427 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.619521 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.619696 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.619777 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342}
W20260929 02:11:46.628875 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:11:46.626912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.11.182.252:38295)
0929 02:11:46.627140 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.627146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.627421 (+ 275us) server_negotiation.cc:407] Connection header received
0929 02:11:46.627602 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.627606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.627693 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.627778 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:11:46.628636 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33748: BlockingRecv error: recv got EOF from 127.0.0.1:33748 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 02:11:46.727682 2531 catalog_manager.cc:1259] Loaded cluster ID: ce5d7cf6958146fcaeed942b1b1afa4b
I20260929 02:11:46.727787 2531 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026: loading cluster ID for follower catalog manager: success
I20260929 02:11:46.730400 2531 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026: acquiring CA information for follower catalog manager: success
I20260929 02:11:46.732271 2531 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a90ce44a9af2497e9286e9cff3714026: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:11:46.765468 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.760149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.11.182.253:45071)
0929 02:11:46.760477 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.760485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.760809 (+ 324us) server_negotiation.cc:407] Connection header received
0929 02:11:46.761056 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.761060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.761144 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.761248 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.761693 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.762414 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.763816 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.764224 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.764362 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.764753 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.764869 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.764971 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.765119 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.765198 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260929 02:11:46.776348 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.770968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.11.182.252:38295)
0929 02:11:46.771129 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.771133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.771442 (+ 309us) server_negotiation.cc:407] Connection header received
0929 02:11:46.771579 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.771583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.771729 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.771836 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.772180 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.772941 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.774270 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.774703 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.774802 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.775285 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.775810 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.775926 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.776023 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.776091 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:11:46.790270 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.782401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.11.182.253:45071)
0929 02:11:46.782649 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.782655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.783679 (+ 1024us) server_negotiation.cc:407] Connection header received
0929 02:11:46.783993 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.784002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.784119 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.784218 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.785005 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.785938 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.788124 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.788802 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.788920 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.789433 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.789604 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.789699 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.789861 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.789944 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:11:46.790750 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.783470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.11.182.252:38295)
0929 02:11:46.783676 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.783682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.784491 (+ 809us) server_negotiation.cc:407] Connection header received
0929 02:11:46.784789 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.784793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.784869 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.784973 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.785564 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.786552 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.788502 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.789102 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.789219 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.789903 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.790034 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.790122 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.790371 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.790464 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:11:46.790946 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:11:46.782234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.11.182.254:46453)
0929 02:11:46.783793 (+ 1559us) server_negotiation.cc:206] Beginning negotiation
0929 02:11:46.783799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:11:46.783817 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:11:46.784006 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:11:46.784014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:11:46.784254 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:11:46.784455 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:11:46.785184 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.786107 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.787985 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:11:46.788655 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:11:46.788785 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:11:46.789343 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:11:46.789900 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:11:46.790397 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0929 02:11:46.790470 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 02:11:46.790561 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1437,"thread_start_us":1321,"threads_started":1}
I20260929 02:11:46.795735 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 2296
I20260929 02:11:46.796878 2359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:46.910197 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 2296
W20260929 02:11:46.931433 2390 connection.cc:535] server connection from 127.0.0.1:38790 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:11:46.931823 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 2371
I20260929 02:11:46.932937 2433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:47.042668 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 2371
I20260929 02:11:47.063411 11995 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkd1a/build/asan/bin/kudu pid 2445
I20260929 02:11:47.064574 2508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:11:47.172905 11995 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkd1a/build/asan/bin/kudu with pid 2445
2026-09-29T02:11:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4393 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9312 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366961 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST